Round-trip engineering apparatus and methods for neural networks

ABSTRACT

Apparatus and methods for high-level neuromorphic network description (HLND) framework that may be configured to enable users to define neuromorphic network architectures using a unified and unambiguous representation that is both human-readable and machine-interpretable. The framework may be used to define nodes types, node-to-node connection types, instantiate node instances for different node types, and to generate instances of connection types between these nodes. To facilitate framework usage, the HLND format may provide the flexibility required by computational neuroscientists and, at the same time, provides a user-friendly interface for users with limited experience in modeling neurons. The HLND kernel may comprise an interface to Elementary Network Description (END) that is optimized for efficient representation of neuronal systems in hardware-independent manner and enables seamless translation of HLND model description into hardware instructions for execution by various processing modules.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 13/239,123, filed Sep. 21, 2011, entitled“ELEMENTARY NETWORK DESCRIPTION FOR NEUROMORPHIC SYSTEMS, which isexpressly incorporated by reference herein.

This application is related to a co-owned U.S. patent application Ser.No. 13/239,163 entitled “ELEMENTARY NETWORK DESCRIPTION FOR EFFICIENTIMPLEMENTATION OF EVENT-TRIGGERED PLASTICITY RULES IN NEUROMORPHICSYSTEMS” filed on Sep. 21, 2011, a co-owned U.S. patent application Ser.No. 13/239,155 entitled “ELEMENTARY NETWORK DESCRIPTION FOR EFFICIENTMEMORY MANAGEMENT IN NEUROMORPHIC SYSTEMS” filed on Sep. 21, 2011, and aco-owned U.S. patent application Ser. No. 13/239,148 entitled“ELEMENTARY NETWORK DESCRIPTION FOR EFFICIENT LINK BETWEEN NEURONALMODELS NEUROMORPHIC SYSTEMS” filed on Sep. 21, 2011, each of theforegoing incorporated herein by reference in its entirety.

COPYRIGHT

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

COMPUTER PROGRAM LISTING APPENDIX ON CD-ROM

The file of this patent includes duplicate copies of compact disc(CD-ROM) with forty six (46) read-only memory files in ASCII fileformat. The file details are presented in Table 1 below. These ASCIIfiles contain lines of code which represent exemplary implementations ofa Computer Program Listing for this disclosure. This CD-ROM and each ofthe files contained thereon and listed in Table 1 is incorporated hereinby reference in its entirety.

TABLE 1 Creation Creation Size Filename Date Time (bytes)synStructure_cpp.txt 6/12/2010 14:51 184 connStructurecopy_cpp.txt7/1/2010 23:15 3,652 connStructure_cpp.txt 9/21/2010 17:13 3,694randConnStructure_cpp.txt 9/21/2010 15:40 1,759 eContainer_cpp.txt6/4/2010  9:47 210 nStructure_cpp.txt 6/24/2010 17:06 226noFactory_cpp.txt 6/23/2010 16:10 348 endO_cpp.txt 6/10/2010  0:31 292nO_cpp.txt 6/30/2010 12:43 237 endObjects_cpp.txt 9/16/2010 18:07 549nContainer_cpp.txt 6/9/2010 22:36 352 main_cpp.txt 9/28/2010 11:57 3,274simple_instances_two_units_xy_end.txt 6/4/2010  9:47 53,710simple_two_units_xy_end.txt 6/4/2010  9:48 6,437 Hashdefs_h.txt10/8/2010 10:39 222 gluSynapse_h.txt 9/16/2010 14:34 1,598synStructure_h.txt 6/22/2010 11:42 1,152 gabaSynapse_h.txt 9/16/201014:34 1,619 inhNeuron_h.txt 7/1/2010 10:12 1,342 layer1exc_h.txt9/16/2010 10:51 1,335 layer1to2exc_copy_h.txt 9/21/2010 17:03 3,321excNeuron_h.txt 7/1/2010 10:12 1,340 layer2inh_h.txt 9/16/2010 10:521,333 layer2to2inh_copy_h.txt 9/21/2010 16:45 2,250 layer2to2inh_h.txt9/21/2010 17:09 1,935 layerSynapse_h.txt 9/28/2010 12:00 3,194eSynStruct_h.txt 6/4/2010  9:47 2,548 endObjects_h.txt 9/16/2010 18:07413 nContainer_h.txt 6/23/2010 20:04 1,593 newNetworkObject_h.txt6/25/2010  9:50 1,869 nGroup_h.txt 7/1/2010 10:12 1,825 endO_h.txt9/17/2010 11:31 9,421 nO_h.txt 7/1/2010 10:08 4,626 eMCN_h.txt 6/9/201022:20 966 noFactory_h.txt 6/23/2010 16:10 941 SpNet_h.txt 9/16/201011:32 3,070 nSpace_h.txt 7/1/2010 10:09 686 eContainer_h.txt 6/10/2010 0:06 1,912 nStructure_h.txt 7/1/2010 10:08 2,642 nSurface_h.txt7/1/2010 10:09 694 connStructure_h.txt 9/24/2010 18:06 5,916randConnStructure_h.txt 7/12/2010  9:58 1,934 randSynStructure_h.txt6/22/2010 18:26 3,121 cNO_h.txt 7/1/2010 10:09 681 layer1to2exc_h.txt9/21/2010 17:05 2,953 testRand_h.txt 6/30/2010 22:53 1,049

BACKGROUND

1. Field of the Disclosure

The present innovation relates to efficient design and implementation ofartificial neural networks.

2. Description of Related Art

Most existing neuronal models and systems include networks of simpleunits, called neurons, which interact with each other via connectionscalled synapses. The information processing in such neuronal systems maybe carried out in parallel.

There are many specialized software tools that may help neuroscientistssimulate models of neural systems. Examples of these tools may includehigh-level implementations such as one or more of NEURON, GENESIS, NEST,BRIAN, and/or other high-level implementations may be designed primarilyfor use by neuroscientists. Such tools may typically require substantialspecialized knowledge, may be cumbersome, and may require customizationin order to achieve efficient performance during simulations that areexecuted using specific software and hardware engines, particularly whenreal-time performance is required, as in autonomous roboticsapplications.

Similarly, low-level implementations such as one or more of assemblylanguages, low-level virtual machine (LLVM) language, Java Bytecode,chip-specific instruction sets, and/or other low-level implementationsmay be designed for efficient hardware implementations on x86, ARM™,and/or other silicon chips. However, such implementations may beunsuitable for parallel simulations of neuronal systems, mostly becausethe silicon chips are not designed for such parallel neuronalsimulations.

Overall, existing approaches have substantial shortcomings as they donot provide sufficient flexibility in designing neural networks, requireexpert knowledge, and/or platform specific customization in order totake advantage of specialized hardware.

Accordingly, there is a salient need for a universal high level networkdescription for defining network architectures in a simple andunambiguous way that is both human-readable and machine-interpretable.

SUMMARY

The present disclosure satisfies the foregoing needs by providing, interalia, apparatus and methods for high level network description forneuromorphic systems.

One aspect of the disclosure relates to one or more systems and/orcomputer-implemented methods for effectuating a neural network using aninstruction set. In one implementation, the method may compriseproviding a representation of the neural network. The representation maycomprise a plurality of instructions of the instruction set. Therepresentation may be compiled into to machine executable format. Theinstruction set may comprise a structured language consistent withEnglish language structure and grammar.

In some implementations, the instruction set may comprise a firstinstruction adapted to cause generation of at least one node within thenetwork. The first instruction may comprise a keyword including one ormore of CREATE, MAKE, PUT, and/or other keywords.

In some implementations, the at least one node may be characterized by aspatial coordinate. The first instruction may comprise a structuredEnglish language statement configured to assign a desired locationwithin the network to the spatial coordinate. The assignment may beperformed as a part of the generation of the at least one node. Thefirst instruction may comprise a keyword including one or more of ON,IN, AT, and/or other keywords.

In some implementations, the generation of at least one node maycomprise generation of a first node and a second node. The instructionset may comprise a second instruction configured to cause a connectionbetween the first node and the second node.

In some implementations, the connection may comprise a synapse, ajunction, and/or another connection.

In some implementations, the generation of at least one node maycomprise generation of a plurality of nodes. The plurality may comprisethe first node and the second node. The second instruction may comprisea Boolean expression configured to select two subsets within theplurality of nodes and/or to cause a plurality of connections betweennodes of one of the two subsets and nodes of another of the two subsets.The second instruction may comprise a keyword including one or more ofCONNECT, LINK, PROJECT, and/or other keywords. The Boolean expressionmay comprise a keyword including one or more of AND, NOT, OR, and/orother keywords.

Some implementations may include a method of programming a computerizedapparatus. In one implementation, the method may comprise representing aneural network using a plurality of instructions of an instruction set.The representation may be compiled into a machine representation forexecution by the computerized apparatus. The instruction set maycomprises a structured language consistent with English languagestructure and grammar.

In some implementations, the network may comprise a plurality ofelements. Individual ones of the plurality of elements may have a tagassociated therewith. The instruction set may comprise a firstinstruction configured to identify a subset of the plurality of elementsbased on the tag.

In some implementations, compiling of the instruction set may beeffected by a database configured to store a plurality of tags. Theplurality of tags may comprise the tag.

In some implementations, the instruction set may comprise a secondinstruction configured to effect assignment of a new tag to the subset.The second instruction may comprise a keyword including one or more ofTAG, ASSIGN, MARK, and/or other keywords.

In some implementations, the instruction set may comprise a Booleanexpression consistent with a structured English representation. Thestructured English representation may enable machine execution of theBoolean expression by an implicit assignment of a logical AND operationbetween two Boolean variables of the Boolean expression. The Booleanvariables may be separated by a separator keyword.

Some implementations may include a method of operating a neuromorphiccomputerized apparatus. In one implementation, the neuromorphiccomputerized apparatus may comprise a nonvolatile storage medium storingone or more of an instruction set, compiling kernel, processing module,and/or other information. The method may comprise providing arepresentation of a neural network to the processing module. Therepresentation may comprise a plurality of instructions included in theinstruction set. The representation may be encoded into hardwareindependent format by the processing module using the kernel. Theinstruction set may comprise a structured language consistent withEnglish language structure and grammar.

In some implementations, the hardware independent format may beconfigured to enable conversion of the representation into a pluralityof machine executable instructions. The plurality of machine executableinstructions may effect operation of the neural network.

In some implementations, the hardware independent format may compriseElementary Network Description (END) format. The machine executableinstructions may include one or more of central processing unit (CPU)instructions, graphics processing unit (GPU) instructions, fieldprogrammable gate array (FPGA) instructions, and/or other machineexecutable instructions.

In some implementations, the machine format may comprise a plurality offunctions configured for execution by the processing module.

In some implementations, the computerized apparatus may comprise aspeech input apparatus. Providing the representation may be effected bya user of the computerized apparatus. The user may speak individual onesof the plurality of instructions such that a given spoken instruction isreceived by the speech input apparatus.

In some implementations, the speech input apparatus may be operablycoupled to the kernel. The kernel may comprise a speech-recognitionmodule configured to translate digitized user speech into one or moreinstructions of the instruction set.

In some implementations, the instruction set may comprise a firstinstruction configured to effect generation of at least one node withinthe network. The at least one node may be characterized by a spatialcoordinate. The first instruction may comprise a structured Englishlanguage statement configured to assign a desired location within thenetwork to the spatial coordinate. The assignment may be effectedsubstantially during creation of the at least one node.

In some implementations, the generation of the at least one node maycomprise generation of a first node and a second node. The instructionset may comprise a second instruction configured to effect a connectionbetween the first node and the second node.

Some implementations may include neuronal network logic. In oneimplementation, the neuronal network logic may comprise a series ofcomputer program steps or instructions executed on a digital processor.The logic may comprise hardware logic. Hardware logic may be embodied inone or more of an ASIC, FPGA, and/or other integrated circuit.

Some implementations may include a computer readable storage mediumhaving at least one computer program stored thereon. The program may beexecutable to implement an artificial neuronal network.

Some implementations may include a system comprising one or more of anartificial neuronal (e.g., spiking) network having a plurality of nodesassociated therewith, a controlled apparatus (e.g., robotic orprosthetic apparatus), and/or other components.

These and other objects, features, and characteristics of the presentdisclosure, as well as the methods of operation and functions of therelated elements of structure and the combination of parts and economiesof manufacture, will become more apparent upon consideration of thefollowing description and the appended claims with reference to theaccompanying drawings, all of which form a part of this specification,wherein like reference numerals designate corresponding parts in thevarious figures. It is to be expressly understood, however, that thedrawings are for the purpose of illustration and description only andare not intended as a definition of the limits of the disclosure. Asused in the specification and in the claims, the singular form of “a”,“an”, and “the” include plural referents unless the context clearlydictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an artificial neural networkcomprising a plurality of nodes and node connections, in accordance withone or more implementations.

FIG. 2 is a block diagram illustrating a neural node type as a networkobject, in accordance with one or more implementations.

FIG. 3A is a block diagram illustrating a node interconnection, inaccordance with one or more implementations.

FIG. 3B is a block diagram illustrating a node interconnectioncomprising a uniform dendrite, in accordance with one or moreimplementations.

FIG. 3C is a block diagram illustrating a non-uniform nodeinterconnection, in accordance with one or more implementations.

FIG. 4 is a block diagram illustrating a public multi-compartment neuron(MCN), in accordance with one or more implementations.

FIG. 5 is an exemplary pseudo code illustrating a public nodedeclaration, in accordance with one or more implementations.

FIG. 6 is a block diagram illustrating a public node interconnectionusing public node definitions, in accordance with one or moreimplementations.

FIG. 7 is a block diagram illustrating a private MCN comprising twoinput interfaces and a single output interface, in accordance with oneor more implementations.

FIG. 8 is a block diagram illustrating a private neuron interconnects,in accordance with one or more implementations.

FIG. 9 is a graphic illustrating a node subset tagging, in accordancewith one or more implementations.

FIG. 10 is a block diagram illustrating spatial tag inheritance, inaccordance with one or more implementations.

FIG. 11 is a block diagram illustrating various exemplaryimplementations of the END engine.

FIG. 12 is a block diagram illustrating node creation using HLND GUIinterface, in accordance with one or more implementations.

FIG. 13 is a block diagram illustrating node subset selection using HLNDGUI interface, in accordance with one or more implementations.

FIG. 13A is a block diagram illustrating node subset selection usingHLND GUI interface, in accordance with one or more implementations.

FIG. 13B is a block diagram illustrating node subset selection usingHLND GUI interface, in accordance with one or more implementations.

FIG. 14 is a block diagram illustrating node selection, tagging, andconnection generation using HLND GUI interface, in accordance with oneor more implementations.

FIG. 15 is a block diagram illustrating neural network computationsusing HLND and END description, in accordance with one or moreimplementations.

FIG. 16 is a block diagram illustrating a computerized apparatus usefulwith the HLND framework, in accordance with one or more implementations.

FIG. 17 is a block diagram illustrating data flow useful with the HLNDframework, in accordance with one or more implementations.

All Figures disclosed herein are ©Copyright 2012 Brain Corporation. Allrights reserved.

DETAILED DESCRIPTION

Implementations of the present disclosure will now be described indetail with reference to the drawings, which are provided asillustrative examples so as to enable those skilled in the art topractice the disclosure. Notably, the figures and examples below are notmeant to limit the scope of the present disclosure to a singleimplementation, but other implementations are possible by way ofinterchange of or combination with some or all of the described orillustrated elements. Wherever convenient, the same reference numberswill be used throughout the drawings to refer to same or similar parts.

Where certain elements of these implementations can be partially orfully implemented using known components, only those portions of suchknown components that are necessary for an understanding of the presentdisclosure will be described, and detailed descriptions of otherportions of such known components will be omitted so as not to obscurethe disclosure.

In the present specification, an implementation showing a singularcomponent should not be considered limiting; rather, the disclosure isintended to encompass other implementations including a plurality of thesame component, and vice-versa, unless explicitly stated otherwiseherein.

Further, the present disclosure encompasses present and future knownequivalents to the components referred to herein by way of illustration.

As used herein, the term “bus” is meant generally to denote all types ofinterconnection or communication architecture that is used to access thesynaptic and neuron memory. The “bus” may be optical, wireless,infrared, and/or another type of communication medium. The exacttopology of the bus could be for example standard “bus”, hierarchicalbus, network-on-chip, address-event-representation (AER) connection,and/or other type of communication topology used for accessing, e.g.,different memories in pulse-based system.

As used herein, the terms “computer”, “computing device”, and“computerized device” may include one or more of personal computers(PCs) and/or minicomputers (e.g., desktop, laptop, and/or other PCs),mainframe computers, workstations, servers, personal digital assistants(PDAs), handheld computers, embedded computers, programmable logicdevices, personal communicators, tablet computers, portable navigationaids, J2ME equipped devices, cellular telephones, smart phones, personalintegrated communication and/or entertainment devices, and/or any otherdevice capable of executing a set of instructions and processing anincoming data signal.

As used herein, the term “computer program” or “software” may includeany sequence of human and/or machine cognizable steps which perform afunction. Such program may be rendered in a programming language and/orenvironment including one or more of C/C++, C#, Fortran, COBOL, MATLAB™,PASCAL, Python, assembly language, markup languages (e.g., HTML, SGML,XML, VoXML), object-oriented environments (e.g., Common Object RequestBroker Architecture (CORBA)), Java™ (e.g., J2ME, Java Beans), BinaryRuntime Environment (e.g., BREW), and/or other programming languagesand/or environments.

As used herein, the terms “connection”, “link”, “transmission channel”,“delay line”, “wireless” may include a causal link between any two ormore entities (whether physical or logical/virtual), which may enableinformation exchange between the entities.

As used herein, the term “memory” may include an integrated circuitand/or other storage device adapted for storing digital data. By way ofnon-limiting example, memory may include one or more of ROM, PROM,EEPROM, DRAM, Mobile DRAM, SDRAM, DDR/2 SDRAM, EDO/FPMS, RLDRAM, SRAM,“flash” memory (e.g., NAND/NOR), memristor memory, PSRAM, and/or othertypes of memory.

As used herein, the terms “microprocessor” and “digital processor” aremeant generally to include digital processing devices. By way ofnon-limiting example, digital processing devices may include one or moreof digital signal processors (DSPs), reduced instruction set computers(RISC), general-purpose (CISC) processors, microprocessors, gate arrays(e.g., field programmable gate arrays (FPGAs)), PLDs, reconfigurablecomputer fabrics (RCFs), array processors, secure microprocessors,application-specific integrated circuits (ASICs), and/or other digitalprocessing devices. Such digital processors may be contained on a singleunitary IC die, or distributed across multiple components.

As used herein, the term “network interface” refers to any signal, data,and/or software interface with a component, network, and/or process. Byway of non-limiting example, a network interface may include one or moreof FireWire (e.g., FW400, FW800, etc.), USB (e.g., USB2), Ethernet(e.g., 10/100, 10/100/1000 (Gigabit Ethernet), 10-Gig-E, etc.), MoCA,Coaxsys (e.g., TVnet™), radio frequency tuner (e.g., in-band or OOB,cable modem, etc.), Wi-Fi (802.11), WiMAX (802.16), PAN (e.g., 802.15),cellular (e.g., 3G, LTE/LTE-A/TD-LTE, GSM, etc.), IrDA families, and/orother network interfaces.

As used herein, the term “synaptic channel”, “connection”, “link”,“transmission channel”, “delay line”, and “communications channel”include a link between any two or more entities (whether physical (wiredor wireless), or logical/virtual) which enables information exchangebetween the entities, and may be characterized by a one or morevariables affecting the information exchange.

As used herein, the term “Wi-Fi” includes one or more of IEEE-Std.802.11, variants of IEEE-Std. 802.11, standards related to IEEE-Std.802.11 (e.g., 802.11 a/b/g/n/s/v), and/or other wireless standards.

As used herein, the term “wireless” means any wireless signal, data,communication, and/or other wireless interface. By way of non-limitingexample, a wireless interface may include one or more of Wi-Fi,Bluetooth, 3G (3GPP/3GPP2), HSDPA/HSUPA, TDMA, CDMA (e.g., IS-95A,WCDMA, etc.), FHSS, DSSS, GSM, PAN/802.15, WiMAX (802.16), 802.20,narrowband/FDMA, OFDM, PCS/DCS, LTE/LTE-A/TD-LTE, analog cellular, CDPD,satellite systems, millimeter wave or microwave systems, acoustic,infrared (i.e., IrDA), and/or other wireless interfaces.

Overview

The present disclosure provides, among other things, a computerized highlevel network description apparatus and methods that may be configuredto define neuronal network architectures in a simple and unambiguousway.

In some implementations, a computerized apparatus may be configured toimplement a High Level Network Description (HLND) kernel. The HLNDkernel may enable users to define neuromorphic network architecturesusing a unified and unambiguous representation that is bothhuman-readable and machine-interpretable.

In some implementations, the HLND format may be used to define nodestypes, node-to-node connection types, instantiate node instances fordifferent node types, dynamically identify and/or select subsets of thenetwork using tags, generate instances of connection between these nodesusing such subsets, and/or other information associated with nodesand/or tags.

The HLND format may provide some or all of the flexibility required bycomputational neuroscientists and may provide a user-friendly interfacefor users with limited experience in modeling neurons.

In some implementations, the HLND kernel may comprise an interface toElementary Network Description (END). The END engine may be configuredfor efficient representation of neuronal systems in hardware-independentmanner and/or may enable seamless translation of HLND model descriptioninto hardware instructions for execution by various processing modules.

In some implementations, the HLND framework may comprise a graphicaluser interface (GUI) configured to enable users to, inter alia, createnodes, select node subsets, connect selected subsets using graphicalactions via the GUI, and/or perform other operations consistent with thedisclosure. The GUI engine may be configured to generate HLNDstatements, which may correspond to the above user actions, withoutfurther input from the user. The HLND framework may be configured toconvert HLND statements into a graphical representation of the networkthat is presented via the GUI. The HLND may include one or morecomponents including (i) the network graphical depiction using the GUI,(ii) the corresponding list of HLND statements, and/or other components.One or more components of the HLND may be configured to consistentlyrepresent the same information about the network, as changes in onerepresentation may be consistently applied to the other representationthereby reflecting some or all of the modifications to the network.

In some implementations, the HLND may be applicable to arbitrary graphstructures (e.g., neural network with arbitrarily complex architecture).

DETAILED DESCRIPTION

Detailed descriptions of the various implementations of the apparatusesand methods of the disclosure are now provided. Although certain aspectsof the disclosure can best be understood in the context of the HighLevel Network Description format used for designing neural networkarchitecture, the disclosure is not so limited and implementations ofthe disclosure may be used for implementing an instruction set that isoptimized for efficient representation of other systems (e.g.,biological or financial) in a hardware-independent manner.

Implementations of the disclosure may be, for example, deployed in ahardware and/or software implementation of a neuromorphic computersystem. In some implementations, a robotic system may include aprocessor embodied in an application specific integrated circuit, whichmay be adapted or configured for use in an embedded application (such asa prosthetic device).

FIG. 1 illustrates one implementation of neuronal network configurationuseful to the disclosure. The network 100 shown in FIG. 1 is comprisedof nodes of different types (the node types 102, 104 in FIG. 1). Asdescribed in detail below, the HLND framework allows users todynamically select an arbitrary subset of network nodes (the subsets106, 108 in FIG. 1) and interconnect nodes of the selected subsets viaconnections (the connections 110 in FIG. 1). Some nodes (e.g., the node104_1) of the network 100 may receive inputs from more than one node(e.g., the nodes 102_1, 102_2 in FIG. 1). Conversely, some nodes (suchas the nodes 102_1, 102_2) may deliver outputs to several nodes, asillustrated in FIG. 1.

HLND Framework Design Overview

According to one or more implementations, the exemplary HLND frameworkmay be configured to facilitate design of neural networks, such asnetwork 100 of FIG. 1. Some implementations may provide an ability todescribe neuronal network of an arbitrary complexity. Someimplementations may facilitate use of pre-defined node and/orpre-defined connection types for the network generation process. Thatis, multiple instances of different node types may be generated,laid-out, and/or connected using multiple instances of differentconnection types. Some implementations may provide a flexible definitionof new node types such that a new node type may comprise animplementation of an elementary network description (END) unit classand/or an implementation of a Network Object (e.g., a layout of nodes, aset of connectivity, and/or a combination of these). The newly definednode types may be used in the network generation process. The ENDframework is described in U.S. patent application Ser. No. 13/239,123entitled “ELEMENTARY NETWORK DESCRIPTION FOR NEUROMORPHIC SYSTEMS”,incorporated supra. Some implementations may provide a flexibledefinition of connection types. A connection type may include animplementation of an END Junction Class, an END Synapse Class, and/orother classes. In one implementation, the newly defined connection typesmay be used in the network generation process. Some implementations mayfacilitate use of universal tags (or labels) for some or all of thenetwork elements including nodes, connections, collections of nodes,and/or other network elements. In some implementations, tags may be usedto identify groups of nodes and/or connections. Tags may be used todynamically select parts of the network. One or more Boolean operations,such as AND, OR, NOT, and/or other Boolean operations may be applied totags. Some implementations may provide an ability to implement a HLNDnetwork using a graphical user interface (GUI). Individual descriptionconstructs may correspond to a user action in the GUI. In someimplementations, models of moderate complexity may be built using a HLNDGUI interface without requiring the use of a keyboard. In someimplementations, the HLND GUI may be operated using a touch screen, alight pen input device, and/or other input technique. Someimplementations may facilitate presentation of HLND statements fordefining network anatomy. Defining network anatomy may include layingout nodes and/or connections in a user-readable (natural English)language to facilitate easy understanding for non-computer expertnetwork designers. Some implementations may provide an ability to useHLND to generate END instances.

Network Definition Methods

Defining a neural network may comprise defining how many and/or whattype of nodes to create, how to lay out these nodes, how to connectthese node instances (e.g., the network layout of FIG. 1), and/or otheroperations. In some implementations, the HLND definition methodcomprises (1) defining new node types and/or connection types for thenew node types, (2) defining a node layout within the network (e.g., howmany and/or what type of nodes to create, and how to arrange these nodeswithin the network that is being created), (3) defining how the nodesconnect to one another, and/or other operations. During neural networkconstruction, the above steps 1-3 may be individually and/orsequentially repeated multiple times. In some implementations, the abovestep 1 may be skipped and pre-defined classes (defining the desired nodetype) may be used instead in defining the network.

In some implementations, a dedicated software package may be configuredto (i) process the HLND statements that define the network and/or (ii)instantiate the network nodes and connections. This processing and/orinstantiation may be subject to one or more constraints including (i)only the defined node types and the defined connection types can beinstantiated and used in the HLND network definition process, (ii) onlythe connections between the existing node instances can be instantiated,and/or other constraints. In other words, only connections correspondingto the defined node instances may be used in the HLND process ofdefining the connections, in accordance with one or moreimplementations. In some implementations, the dedicated software packagemay comprise the END engine, which may be configured to generate the ENDinstance of network models, as described in a co-owned U.S. patentapplication Ser. No. 13/239,123 entitled “ELEMENTARY NETWORK DESCRIPTIONFOR NEUROMORPHIC SYSTEMS”, incorporated supra.

Defining Node Types

The definition of a node type may provide the implementationinstructions of the node, which may be configured to instruct a networkprocessing apparatus to perform specific steps during instantiation ofthe node in accordance with the node type. In some implementations, thenode definition may further specify internal implementation of the nodetype (for example, specify the dynamics of the neuron type). In one ormore implementations, the node definition may comprise definitions ofinput ports and/or output ports for the node

In some implementations, the node type may be defined as a Simple Node,where the node definition specifies the “internals” of that node.Internals of a node may include the implementation of an END unit (i.e.,a neuron) and/or an END implementation of a neuronal compartment.

In some implementations, the node type may be defined as a complexNetwork Object, which may provide instructions on how to instantiatepre-defined node types, instructions on how to connect the nodes, and/orother instructions. Instructions on how to connect nodes may include aHLND description of a network of an arbitrary complexity, an algorithmconfigured to specify details of node and/or connection instancegeneration, and/or other instructions. It will be appreciated by thoseskilled in the arts that the term Network Object may be used to describeany network that is implementable using the HLND framework.

Within the HLND framework description, individual node types maycomprise intra-node connections and may define interface (or interfaces)for incoming connections and/or outgoing connections.

Node Type as an END Unit

In some implementations, the END Unit class may be generated using nodeimplementation details (e.g., update rule, event rule). See, e.g., U.S.patent application Ser. No. 13,239,123 for additional detail related tothe END Unit classes.

Node Type as a Network Object

In some implementations, the definition of a network object may beconfigured similarly to the definition of a network, with the maindifference being that the network object is reusable. That is, multipleinstances of the network object may be instantiated. Some or allelements of the network object (e.g., units, tags, and/or otherelements) may be scoped—that is, they may have a finite lifetimeassociated with a particular scope. In some implementations, networkobjects may be configured to provide an I/O interface that may be usedto connect the network object with other nodes. A network object may besimilar to a building block in a Simulink® model (see, e.g.,http://www.mathworks.com/products/simulink/index.html), a p-cell in acomputer aided design (CAD) software, a function/class in C++ languagecode, and/or other programming elements.

In some implementations, a network object may be allowed to usepre-defined nodes that are defined externally (i.e., outside the scopeof this node type). Thus, the parent node (i.e., the network object) andthe child node(s) (i.e., the node types used in the network object) maynot comprise nodes of the same type. In other words, the definition ofan ‘X’-type node may not instantiate ‘X’ type nodes, according to someimplementations.

Referring now to FIG. 2, one exemplary implementation of a networkobject is illustrated and described in detail. The network object 200may comprise one or more of a network definition, specification of theobject input/output (I/O) interface, and/or other information.

The network definition may specify one or more steps of the objectinstance generation. The network definition may be implemented via astandard HLND network definition. A standard HLND network definition maycomprise an instantiation and/or layout of nodes that specifies thenumber of instances of each pre-defined node and/or their spatialarrangement using pre-defined distribution functions. A standard HLNDnetwork definition may comprise a connectivity description, which maydefine connectivity between nodes. In some implementations, theconnectivity description may define and/or use spatial projection forthe node. By way of non-limiting example, defining spatial projectionfor a node may include defining axonal and/or dendritic projections forthe nodes (e.g., (i) dendritic spread, (ii) distribution of synapticboutons, (iii) distribution of axon terminals), defining how the axon ofa node (e.g., model neuron) connects to the dendrite of another node(e.g., model neurons), and/or defining other information associated withspatial projection for a node.

By way of non-limiting example, the standard HLND network definition maybe used in (i) defining a specific layout of pre-defined nodes; (ii)defining a multi-compartmental neuron (e.g., a collection of pre-definedEND units connected with pre-defined END junctions); (iii) defining anarbitrarily complex network comprising a plurality of neurons; and/ordefining other information associated with the standard HLND networkdefinition. In some implementations, a network may comprise synapsesand/or junctions.

In some implementations, the definition of the network may compriseinstance generation with an algorithm, which may be configured todescribe one or more steps of Network Object instance generation, usingthe above Object definition. By way of non-limiting example, thealgorithm may include one or more of (i) an algorithm that usespre-defined node types and/or defines the instance generation process ofsuch node types; (ii) an arbitrary algorithm that uses pre-defined nodetypes and/or connection types and/or define the instance generationprocess of such node and/or connection types; (iii) an algorithm thatdefines a dendritic tree; and/or other algorithms.

The above exemplary algorithms may utilize multiple pre-defined END unittypes implementing neural compartments and/or pre-defined END junctiontypes designed to connect such compartments with the algorithm definingthe layout of compartments and the connection between them. (See, e.g.,Cuntz H., Forstner, F., Borst. A, and Häusser, M. (2010), “One Rule toGrow Them All: A General Theory of Neuronal Branching and Its PracticalApplication. PLoS Computational Biology, 6(8), incorporated herein byreference in its entirety). The I/O interface may specify aninput/output connection implemented for the network object.

Definition of Connection Types

Within HLND, definition of a connection type may provide the necessaryimplementation details (e.g., pre-event rules, post-event rules, updaterules, and/or other details) to generate a connection that comprises oneor both of (i) an END synapse or (ii) an END junction, in accordancewith one or more implementations.

Instantiating Nodes

In some implementations, the HLND may define rules that govern nodeinstantiation. The HLND node instantiation instructions may besubsequently provided to a software package (e.g., the END kernel) thatinterprets the instructions and instantiates the appropriate nodes.According to some implementations, during instantiation, some or allnode types may be treated equally, regardless of whether they are simplenodes (e.g., the END implementation of a neuron) or network objects(e.g., the entire network description). In some implementations, thefollowing information may be needed to instantiate and lay out nodes:(i) node type to be instantiated, (ii) number of instances of the nodetype to be instantiated, and/or other information.

In a basic form, an HLND instantiation statement may create n instancesof a given node type using default definitions corresponding to the nodetype. During instantiation, additional parameters may be used to, interalia, (i) set parameters used to initialize the instantiated node types,(ii) set how the instantiated nodes are laid out in space (e.g., howposition tags are assigned), (iii) add additional tags to the new nodeinstances, and/or perform other operations associated with HLNDinstantiation. Within HLND, a defined node type available within thescope of operation may be instantiated without restriction.

In some implementations, position coordinates (i.e., spatial tags) maybe assigned to the generated node instances during node instantiation.In order to implement this functionality, the HLND framework may supportusage of predefined distribution functions that assign spatial tags toeach instantiated node. Such a distribution function may be configuredto sample n points from a given probability density function. By way ofnon-limiting example, the HLND statement:

-   -   Uniform(n, boundary parameters)        may describe sampling of n points that are uniformly distributed        within a space range, defined by the boundary parameters        argument. Similarly, the HLND statement:    -   Normal(n, sigma, boundary parameters)        may describe sampling of n normally distributed points within a        space range specified by the boundary parameter.

In addition to the unique id tags that individual generated nodes mayhave, extra tags may be optionally assigned during the instantiationprocess and may be used to identify the set of newly instantiated nodes.In some implementations, special reserved tags (e.g., “IN”, “OUT”, orother special reserved tags) may be used to specify that the generatedunits are input or output interfaces for the network, therefore makingthese nodes accessible (readable and/or writable) from outside.Exemplary HLND calls are shown in the Listing 1 below:

Listing 1. a) create 100 of the ‘exc’ node types (assuming that the‘exc’ END class/network object exists)   example1_exc_neurons = (100,‘exc’) b) create 200 of the ‘exc’ node types and distribute them usingthe given pdf:   example2_exc_neurons = (200, ‘exc’, _exc_parameters_,  pdf) c) create 1 node of a network object type that implements theretina (assuming that the ‘retina’ network object type has beenimplemented previously):   example3_retina = (1, ‘retina’,_retina_parameters_)Morphology/Extensions

In some implementations, morphology/extensions may be used duringconnection instantiation. The morphology definition described withrespect to FIG. 2 supra, specifies how an instantiated node may projectand/or extend into network space. That is, the morphology may definespatial extent where an instantiated node is allowed (i) to “receive”incoming connections from, and/or (ii) to “send” outgoing connectionsto. Note that, according to some implementations, an addition of a nodeextension may not alter the size and/or the location of the node.Instead, extensions may enable a node to “search” for other nodes duringinstantiation of node interconnections. In other words, an extension mayprovide an additional “view” of a node, which may be used during theprocess of connecting nodes.

In some implementations, connecting nodes using spatial tags may only beallowed if the nodes are overlapping. Nodes may have a zero extension bydefault. In some implementations, only co-located nodes may beconnected. In order to extend node connectivity, non-zero node input(dendrite) and node output (axon) projections may be defined. In someimplementations, these projections may be used when connecting any twonodes. For example, the output projection of one node may be connectedto the input projection of another node.

To create an extension, some or all of the following information may berequired, in accordance with one or more implementations: (1) sourcetags used to identify the nodes for which the extension is created; (2)extension tags that are used to identify the extensions to be created;and/or (3) define I/O point distribution to define the space/extensionwhere the node can receive incoming connections, and define the spacewhere the nodes can have outgoing connections.

For incoming extensions, the distribution of receiving terminals may bespecified. The distribution of receiving terminals may be analogous todendritic spread, and/or distribution of synaptic boutons in case of aneuron. For an outgoing extension, the projection spread may bespecified. The projection spread may be analogous to axon terminals.

For the distribution of receiving terminals and spread of projections,HLND may support predefined functions (e.g., bounded Gaussian and/oruniform distribution). In general, arbitrary density functions may beused.

Exemplary HLND calls are shown in the Listing 2 below:

Listing 2. example1_exc_neurons_axon = (‘example1_exc_neurons’, pdf1)example2_exc_neurons_dendrite = (‘example2_exc_neurons’, pdf2)In another approach, connections are instantiated without the use ofmorphology/extension.Connecting Nodes

The HLND connection statement may comprise instructions configured toinstantiate connections (of a given connection type) from one set ofnodes to another set of nodes. In some implementations, some or all ofthe following information may be required in order to enable thesenode-to-node connections: (1) “from subset”, (2) “to subset”, and/or (3)“connection type”. A from subset may include a node subset selected bytags that uniquely identify source nodes/extensions (e.g.,nodes/extensions that a connection will originate from). A to subset mayinclude a node subset selected by tags that uniquely identifydestination nodes/extensions (e.g., nodes/extensions that a connectionwill terminate at). A connection type may include the connection typeused to connect <From Subset> nodes/extensions to <To Subset>nodes/extensions.

In some implementations, the HLND connection statement may directconnection instantiation with a given connection type from all available<From Tags> nodes to all available <To Tags> nodes. According to someimplementations, the connection parameters may be used to filter outconnections. That is, filter constrains may be applied to the some orall possible <From Tags> to <To Tags> connections. Therefore, a subsetof possible <From Tags> to <To Tags> connections may be instantiated,which may allow instantiation of arbitrary connection maps from <FromTags> to <To Tags>. In some implementations, connections may beexpressed as function calls. In some implementations, connections may beexpressed using a table. In some implementations, a connection filtermay be configured to generate all-to-all connections, where all of the<From Tags> are connected to all of the <To Tags>.

As an aside, two notation formats <From Tags> and <From Subset> may beused in various implementations, as both notations may cause the HLND togenerate connections for a subset. For example, notation <‘FromTag1’ AND‘FromTag2’> may describe a collection of nodes (e.g., Collection1) thathave both tags ‘FromTag1’ and ‘FromTag2’. Accordingly, notation:<‘Collection1’> may be used instead while producing the same results.

By way of non-limiting example, the following connection statements maybe employed:

  exc2exc = (pre=’example1_exc_neurons_axon’,post=’example2_exc_neurons_dendrite’, 100 connections / cell,SynType=’GLU’, _other_parameters_).

In some implementations, the HLND connections statement may beconfigured to enable parameterized connection establishment, such thatparameters may be passed to the connection type to set the connectionvariables in the connection instances. In some implementations, theconnection parameters may be used to set the weights of the synapticnode connection. In some implementations, the node information (e.g.,the position of the from-node and the to-node) may be used to set upconnection weights based on the distance between the nodes.

By way of non-limiting example, one or more of the following connectionstatements may be employed: (1) connect each <From Nodes> node to N <ToNodes> node; (2) connect to each <To Nodes> node N <From Nodes> node;and/or (3) randomly sample N from all possible connections.

In some implementations, nodes may comprise position tags and/or mayhave zero default extension. Such nodes may connect to co-located nodes.Connecting nodes with spatial tags may require overlap so thatoverlapping nodes may be connected.

Referring now to FIG. 3A, an exemplary implementation of an HLNDframework node interconnection is shown and described in detail. Thenetwork 300 of FIG. 3A may comprise a group of A nodes 302 and a groupof B nodes 304. For clarity, the network 300 may be configured using aone-dimensional configuration, so that nodes with matching X-coordinates306 (i.e., the node index i=1:7 in FIG. 3A) are allowed to be connectedvia the connections 308. Specifically, a node a_i from the node group302 may be allowed to be connected to the respective node to b_j of thenode group 304 when i=j, such as, for example the nodes 312, 314,respectively, in FIG. 3A. Several possible connections are illustratedby solid lines 308.

In some implementations, such as illustrated in FIGS. 3B-3C, nodeextensions may be added to the nodes of the node groups, respectively,in order to implement complex connections and to enable more flexibleconnection mapping. The node extensions may be used to, inter alia, mapspatial coordinates of the source nodes (e.g., the nodes A of the group322 in FIG. 3B) to the receiving nodes (e.g., the node 324 in FIG. 3B).The extensions may be used to define the probability density function ofpotential connections, such as in the exemplary implementationillustrated in FIG. 3C.

FIG. 3B illustrates an exemplary implementation of node to nodeconnection configuration comprising node extensions. The network 320 ofFIG. 3B may comprise the group 322 of A nodes a_1:a_7 and single B node324. The network 320 of FIG. 3B may be configured using a singledimension so that the nodes with matching X-coordinates 306 may form aconnection. The B node 324 of FIG. 3B may comprise a uniform Dendriteextension 330 of the dimension 332. Individual nodes of the A node group322 (such as the node 322_1) may comprise a uniform Axon extension 326of the dimension 328. The axon dimension 328 may be smaller than thedendrite dimension 332. The term “uniform extension” may be used todescribe the uniform probability distribution of connections foroverlapping extensions (Axon or Dendrite in this example). That is, nodeconnections may be equally likely provided their extensions overlap,according to some implementations. For a one dimensional extension(e.g., the extension 330 in FIG. 3B), this may correspond to a uniformconnection probability along the extent 332 of the extension 330. For amulti-dimensional extension, the uniform extension may correspond to auniform connection probability in all dimensions.

The network configuration illustrated in FIG. 3B may allow connectionsbetween the node a_i of the A group 322 to the b_1 node 324 when theaxons 326 _(—) i overlap with the spatial dimension 332 of the dendrite330. As illustrated in FIG. 3B, the nodes a_3, a_4, a_5 may be connectedto the node b_1 via the connections, depicted by solid arrows 308 inFIG. 3B. Inactive (e.g., not allowed) connections 318 between other Anodes are depicted by dashed arrows in FIG. 3B. The tags <‘axon’> and<‘dendrite’> may refer to another ‘view’ of the node.

As illustrated in FIG. 3B, the following extension may be constructed:(i) a uniform circle extension (denoted as the Dendrite) to the B node330 in FIG. 3B; (ii) a uniform circle extension (denoted as the Axon) toall A nodes in FIG. 3B; and/or (iii) Connect <A Axon> to <B Dendrite>.The A to B node connection may be possible and may be instantiatedbecause both the sending and receiving extensions may be uniform. Inthis example, the connect statement is looking for extensionoverlaps—that is, whether the <Axon> extension of a node <A> overlapswith the <Dendrite> extension of a node <B>.

Within the HLND framework, nodes may comprise different views, such as,for example, Axons or Dendrites. Node tags ‘axon’ or ‘dendrite’ may beused in HLND to refer to the same node. The Axon/Dendrite may havedifferent spatial properties.

FIG. 3C illustrates an exemplary implementation of node-to-nodeconnection configuration comprising non-uniform node extensions. Thenetwork 338 depicted in FIG. 3C may comprise the group 322 of A nodesand the B node 344. Individual ones of the A nodes may comprise auniform extension 326. The B node 344 may comprise a non-uniformextension 340. The term non-uniform extension may be used to describethe node extension (Axon or Dendrite) that has a non-uniform connectionprobability distribution across at least one dimension of the extension.For a one dimensional extension (e.g., the extension 340 in FIG. 3C),this corresponds to a non-uniform extension connection probabilitydistribution along the extension extent 342. In some implementations,the extension connectivity parameter P may comprise a connectionlikelihood, which may be characterized by a probability function as afunction of the extension extent 342.

The connectivity profile of the non-uniform extension 340 (see FIG. 3C)may be configured using a shape of the Gaussian distribution and/orother distributions. The extension 340 may be centered at the node 344.The extension 340 may be characterized by a certain variance σ² and aradius 348.

Node connections via non-uniform extension are illustrated in FIG. 3C.The network configuration illustrated in FIG. 3C may allow connectionsbetween the node a_i of the A group 322 to the b node 344 when the axons326 _(—) i overlap with the spatial dimension 342 of the non-uniformdendrite 340. Possible connections in the network 338 include a_3, a_4,. . . , a_10 axon to b dendrite, as a_1 axon, a_2 axon, and a_11 axon donot overlap with the dendrite 340 spatial dimension 342.

The non-uniform extensions (e.g., the extension 340 in FIG. 3C) may biasthe selection of connections towards axons that overlap with the highestprobability area of the non-uniform dendrite. While the extensiondimension overlap may be used to identify all of the possibleconnections, the sampling of the possible connections may follow theconnectivity profile (probability) of the extensions. When selecting asubset of the possible connections the connectivity profile (shape) ofthe extension (that describes connection likelihood) may need to betaken into account. By way of non-limiting example, when connecting asingle arbitrary <A axon> 326 to the <b dendrite> 340 (see FIG. 3C), themost likely outcome may be a connection between the node a_6 and thenode 344, rather than connections between nodes a_1, a_10 to the node344.

In some implementations, the HLND exemplary operation sequence forconnecting node populations using non-uniform extension may be: (1) addGaussian extensions centered at the node with a fixed radius r1 to Bnodes and tag these extensions as <Dendrite>; (2) add uniform extensionswith a fixed radius r₂ to A-nodes and tag these extension as <Axon>; and(3) connect N random <A Axon>s to <B Dendrite>s. The possibleconnections may be where the <Axon> extensions of <A> nodes overlap withthe <Dendrite> extensions of <B> nodes. Instantiated connections maycorrespond to the highest connectivity. In some implementations, thehighest connectivity may be determined based on a product of theGaussian and the uniform functions.

I/O for Network Objects

In some implementations of the HLND framework, network objects maycomprise one or more members and an Input/output (I/O) interface. TheI/O interface may specify how to interface (e.g., establish connections)with other elements of the network. In some implementations, the membersof the network object may comprise nodes. In some implementations, themembers of the network object may comprise nodes and connections. TheI/O interface may define how the object members are accessible fromoutside the scope of the network object. During definition, individualmembers of the network object (and their values) may be declared aspublic or private. Private members may not be visible (i.e., directlynot accessible) to external network elements outside the network object.The private object members may be accessible via the I/O interfacedefined for that member. That is, private members of a network objectmay not be visible from outside the scope of the network object. In thiscase, the I/O interface may be required to implement connections.

In some implementations, the network objects may be defined as ‘open’.Members of a network object defined as open may be public and visiblefrom outside the scope of that network object. This may alleviate arequirement to publish the I/O interface.

Public members of the network object may be visible and/or accessible byexternal elements for input and/or output connections. In someimplementations, members of the network objects may be scoped bydefault. That is, some or all variables within the network object may bescoped inside the network object. Members of multiple instances of thesame network object type may not interfere when these members use thesame tags.

In some implementations, a network object may be defined as a ‘macro’. Anetwork object defined as a macro may not be treated as scoped object.Such macro definition may allow some or all of the variables definedwithin the macro object to be accessible and/or visible by externalelements.

By way of non-limiting example, node_a may be a public member of networkobject NO1, node_b (that is not a member of NO1) may connect directly tothe node_a, and/or receive connections from the node_b. A member node_ain NO1 may be accessed by using the tags <NO2> and <node_a>, and/or withNO1.node_a scoped notation. A node_c that is a private member of networkobject NO2 may not be visible and/or accessible from outside unless theI/O interface is defined for the node_c member of the NO2, according tosome implementations. An external node (that is not member of the NO2)may not connect to and/or receive a connection from the node_a memberdirectly. In other words, a public member of a network object may beaccessed using tags and/or other publicly available information.

An exemplary implementation of the HLND network object, illustrating apublic multi-compartment neuron (MCN), is shown and described inconnection with FIG. 4. The MCN neuron 400 may comprise one or morepublic nodes, which may include dendritic compartments 404 and somacompartment 402. The term public may refer to members of the networkobject (e.g., the compartments 402, 404) that are visible outside thescope of their definitions (e.g., outside the MCN 400). Individualcompartments 404 may be assigned two tags, which may include DENDRITE,COMP, and/or other tags. Compartment 402 may be assigned three tags,which may include DENDRITE, COMP, SOMA, and/or other tags. Thecompartment 402 and individual compartments 404 may be connected viajunction connections 406.

By way of non-limiting example to illustrate functionality of publicnetwork elements in accordance with some implementations, two instancesof public MCN neuron 400 may be considered. One instance may be taggedas the ‘neuron_a’, and another instance may be tagged as the ‘neuron_b’.The notation <neuron_a AND soma> collection may refer to the member 402of MCN with the SOMA Tag in the neuron_a instance. The <neuron_b ANDSOMA> collection may refer to the member 402 of MCN with the SOMA Tag inthe neuron_b instance. As some or all members 402, 404 of the MCN<neuron_a> may be public, they may be visible to outside entities (e.g.,the MCN <neuron_b>), which may enable direct connection of the <neuron_aAND SOMA> to the <neuron_b AND DENDRITE AND COMP>.

In some implementations, connections from <neuron_a AND SOMA> to<neuron_b AND SOMA> may be instantiated using a given connection type.

As will be appreciated by those skilled in the arts, the above notationis exemplary and various other notations may be used in order toidentify, select, and/or access node members using their tags.

FIG. 5 illustrates connection instantiation between public neurons A andB (denoted by the designators 500 and 520 in FIG. 5, respectively), inaccordance with one or more implementations. Individual members 504 ofthe node A and individual members b 524 of the node B may be public.Individual ‘a’ members 504 may be connected to individual ‘b’ members524. An example is illustrated in FIG. 5 where the out projection 508may overlap with multiple in projections 526 of the nodes 524, so thatthe connection 508 may be established between the single a node 504 andone or more b-nodes 524.

An exemplary pseudo-code, corresponding to the implementationillustrated in FIG. 5, is presented in FIG. 6. The statements 600 and620 in FIG. 6 may be configured to generate the node instances 504 and524 of FIG. 5, respectively. The statements 606 and 626 in FIG. 6 may beconfigured to define the out projections 506 and/or the in projections526 of FIG. 5, respectively. The last statement 610 may be configured todefine the connection 508.

FIG. 7 illustrates an exemplary implementation of a network objectcomprising a private multi-compartment neuron. The private MCN neuron700 may comprise one or more private compartments 704 and/or a privatecompartment 702. The term ‘private’ may refer to network members (e.g.,the compartments 702 and 704) that are not visible outside thedefinition scope of the respective network object (e.g., outside the MCN700). Individual compartments 704 may be assigned two tags, which mayinclude DENDRITE, COMP, and/or other tags. The compartment 702 may beassigned three tags, which may include DENDRITE, COMP, SOMA, and/orother tags.

By way of non-limiting illustration, two instances of the private MCN700 type may be considered. One instance may be tagged as “neuron_a”,and another instance may be tagged as “neuron_b”. As the MCN 700 type isdefined as private, the MCN 700 members (e.g., the <neuron_a ANDDENDRITE & COMP>, <neuron_a AND SOMA>, <neuron_b & DENDRITE & COMP>,<neuron_b & SOMA>, <SOMA> collections) may be invisible from outside theMCN 700. Directly connecting the <neuron_a AND SOMA> to the <neuron_bAND DENDRITE AND COMP> may not be permitted for the node configurationtype illustrated in FIG. 7, in accordance with some implementations. Inorder to enable neuron external connectivity, the MCN 700 definition maycomprise input ports 714 and 716, and the output 718 port, may be usedto specify the I/O interface for the MCN node type. The input interfaceof the MCN 700 may comprise direct internal connections to the privatemembers of the MCN 700. Direct internal connections to the privatemembers of the MCN 700 may include the connections 726, 728, and 730from the input interface IN1 714 to the private members 704 with tags“DENDRITE” and “COMP”. Direct internal connections to the privatemembers of the MCN 700 may include the connection 720 from the inputinterface IN2 716 to the private member 702 with the tag “SOMA”. Thelink 722 may connect the private member 702 to the Output interface 718,which may allow the node 702 to be used for outgoing connections and/orfor incoming connections. The neuron_a.OUT may be configured to belinked/connected to the neuron_b.IN1. The neuron_b.OUT may be configuredto be linked/connected to the neuron_a.IN2.

FIG. 8 illustrates an exemplary connection instantiation between privatenetwork objects A and B (denoted by the designators 800 and 820 in FIG.8A, respectively). Individual ones of the members 804 of the node Aand/or individual ones of the members 824 of the node B may be privateand, hence, may be inaccessible by outside instances. That is, the ‘a’node members 804 may be inaccessible by the ‘b’ node members 824 andvice versa, according to various implementations. In order to enable thenode instances 800 to generate external connections to nodes 820, I/Ointerfaces for the private node members may be required. In someimplementations, the I/O interfaces may comprise input/output ports(e.g., the I/O ports 714, 716, and 718, described with respect to FIG.7, supra). A private node instance (e.g., the instance 800) may comprisea large number (e.g., millions) of other private members. A private nodeinstance may provide an outgoing interface 812 for those members thatwill be used for outgoing connections.

Conversely, while individual ones of the members 824 of the nodeinstance 820 may be kept private, an input interface 822 may be used tospecify how the node members 824 connect to the input port 822,according to some implementations. Although a single port is illustratedin the implementation of FIG. 8A, this is not intended to be limiting asmultiple uniquely tagged input/output ports may be used in someimplementations.

As the output interface 812 of the node instance 800 and the inputinterface 822 of the node instance 820 may be exposed and/or visible toexternal network elements, a link/connection 810 may be establishedbetween the node instances of network objects 800 and 820 by using inand out interfaces 812 and 822, in a manner that is similar to theconnection establishment described with respect to FIG. 5, FIG. 7,supra.

FIG. 8B presents an exemplary pseudo-code, corresponding to theconnection establishment implementation illustrated in FIG. 8A. Thestatement 830 in FIG. 8B may create the node instances 804 in FIG. 8Aand may expose the output port 812. The statement 850 in FIG. 8B maycreate the node instances 824 in FIG. 8A. Statements 856 and 852 in FIG.8B may define the in projections 826 of ‘b’ nodes 824 of FIG. 8A and/orthe in interface 822 of FIG. 8A, respectively. The statement 858 maydefine the connection between input interface and ‘b’ members.

In some implementations, a third network object 860 may create theinstances inst_a and inst_b of types A 800 and B 820, respectively,and/or may connect the inst_a instance to the inst_b instance using thea_out port of the node instance 800 and the in port of the node instance820. The exemplary HLND definition steps, shown in FIG. 8B, may include(1) creating an instance of A (inst_a), (2) creating an instance of B(inst_b), and/or connecting and/or linging inst_a to inst_b.

As the members of the node instances A and B may be private, the objectC may not directly connect the members of the instance A to the membersof the instance B. Instead, the object C may use the exposed portsinst_a.a_out to the inst_b.in. In the connection, declaration 868 mayuse the equal sign notation to denote that the inst_b.in is assigned to(e.g., is the same as) the inst_a.a_out. In some implementations, theHLND compiler may use the definition 868 and a definition of the privatenode B members to resolve the connection between the virtual b.in portto the actual members of the node B 820 by linking the inst_b.in withthe corresponding member(s) of the node instance A 800—that is,indirectly establishing the connection between the inst_a (of A) and theinst_b (of B). As illustrated in FIG. 8A, the node type 800 and 820 mayspecify projection extensions and/or synapse/connection types that maybe required to effect the node to node connection.

Tags

In accordance with some implementations, individual elements of thenetwork (i.e., nodes, extensions, connections, I/O ports) may beassigned at least one unique tag to facilitate the HLND operation anddisambiguation. The tags may be used to identify and/or refer to therespective network elements (e.g., a subset of nodes of the network thatis within a specified area).

In some implementations, tags may be used to form a dynamic grouping ofthe nodes so that these dynamically created node groups may be connectedwith one another. That is, a node group tag may be used to identify asubset of nodes and/or to create new connections within the network, asdescribed in detail below in connection with FIG. 9. These additionaltags may not create new instances of network elements, but may add tagsto existing instances so that the additional tags are used to identifythe tagged instances.

FIG. 9 illustrates an exemplary implementation of using additional tagsto identify tagged instances. The network node population 900 maycomprise one or more nodes 902 (tagged as ‘MyNodes’), one or more nodes904 (tagged as ‘MyNodes’ and ‘Subset’), and/or other nodes. The darktriangles in the node population 900 of FIG. 9 may denote the nodes 902tagged as ‘MyNodes’, while the black and white triangles may correspondto a subset of nodes 904 that are tagged as ‘MyNodes’, and ‘Subset’.

Using the tag ‘MyNodes’, a node collection 910 may be selected. The nodecollection 910 may comprise individual ones of nodes 902 and/or 904(see, e.g., FIG. 9). The node collection 920 may represent nodes taggedas <‘MyNodes’ NOT ‘Subset’>. The node collection 920 may compriseindividual ones of the nodes 902. The node collection 930 may representthe nodes tagged as ‘Subset’. The node collection 930 may compriseindividual ones of the nodes 904 (see, e.g., FIG. 9).

In some implementations, the HLND framework may use two types of tags,which may include string tags, numeric tags, and/or other tags. In someimplementations, the nodes may comprise arbitrary user-defined tags.Numeric tags may include numeric identifier (ID) tags, spatial tags,and/or other tags.

Upon instantiating a node, the instantiated node may have a string tag(the node type) and a unique numerical tag (the unique numericalidentifier). In some implementations, position tags may be assignedduring the instantiation process.

Operations on Tags

As shown in FIG. 9B, the tags may be used to identify a subset of thenetwork. To implement this functionality, one or more Boolean operationsmay be used on tags. In some implementations, mathematical logicaloperations may be used with numerical tags. The < . . . > notation mayidentify a subset of the network, where the string encapsulated by thechevrons < > may define operations configured to identify and/or selectthe subset. By way of non-limiting illustration, <‘MyTag’> may selectindividual nodes from the network that have the tag ‘MyTag’; <‘MyTag1’AND ‘MyTag2’> may select individual members from the network that haveboth the ‘MyTag1’ and the ‘MyTag2’ string tags; <‘MyTag1’ OR ‘MyTag2’>may select individual members from the network that has the ‘MyTag1’ orthe ‘MyTag2’ string tags; <‘MyTag1’ NOT ‘MyTag2’> may select individualmembers from the network that have the string tag ‘MyTag1’ but do nothave the string tag ‘MyTag2’; and <‘MyTag1’ AND MyMathFunction(SpatialTags)<NumericalValue1> may select individual nodes from the network thathave the string tag ‘MyTag1’ and the output provided by MyMathFunction(as applied the spatial coordinates of the node) is smaller thanNumericalValue1. Note that this example assumes the existence of spatialtags, which are not mandatory, in accordance with variousimplementations.

Tag Inheritance

In some implementations, the HLND framework may comprise hierarchicaltag inheritance. In some implementations, individual members that areinstantiated within a network object may inherit string tags of itsparent. For example, individual members of a network object with tags‘ParentTag1’ and ‘ParentTag2’ may comprise the tags ‘ParentTag1’ and‘ParentTag2’ in addition to the member-specific tags assigned, forexample, during member instantiation.

In some implementations, member spatial tag data may refer to localcoordinates (referenced relative to the space defined by the networkobject) of the member. In some implementations, global coordinates(referenced relative to the space of the entire network) may be inferredfrom the nested structure of network objects and/or members.

FIG. 10 illustrates an exemplary implementation of spatial taginheritance. A network object B (not shown) may instantiate a singleinstance of the node of type C, e.g., the node_c at position (1,1),denoted as 1002 in FIG. 10. A network object A (not shown) mayinstantiate two instances of the node type B, e.g., the node_b_1 atposition (1,1) and the node_b_2 at position (1,2), denoted as 1004 and1006, respectively, in FIG. 10. The coordinates of the node_c may bereferenced to the scope of the node_b_1. The coordinates of the node_b_2may be set to (1,1). The coordinates of the node_b_1 and the node_b_2referenced to the scope of the node_a may be set as (1,1) and (1,2),respectively. The coordinates of the node_c in the node_b_1 referencedto the scope of the node_a may be determined as (1,1)+(1,1)=(2,2). Thecoordinate of the node_c in the node_b_2 referenced to the scope of thenode_a may be determined as (1,2)+(1,1)=(2,3). Note that the notationnode_c, node_b_1, node_b_2 and node_a may be used to identify theinstantiated objects of type C, B, and A, respectively.

HLND tag properties and/or characteristics, according to one or moreimplementations, may be summarized as: tag types may include string tagsand numerical tags; numerical tags may comprise the numericalidentifier; Boolean operations may be used on tags; math functions maybe allowed on numerical tags; optional spatial and string tags may beassigned; individual node instances may comprise a unique numericalidentifier tag; string tag inheritance may be hierarchical; spatial tagsmay refer to local coordinates; global tag coordinates may be inferredfrom the nested structure of nodes; and/or other properties and/orcharacteristics.

Tag Implementation

In some implementations, the HLND framework implementation of tags maybe configured to require the following functionality: (i) an interfaceto tag-data generator and data handler; and (ii) implementation ofnested objects so as to enable creation of complex network objects fromany number of existing network objects.

In some implementations, the tag data handler may be implemented using adatabase, such as, e.g., MySQL. Instances of network objects may begenerated using arbitrary string tags. In some implementations, thenetwork objects may be generated using position tags as well as thestring tags. Tag data may be placed into a database. Additional networkdata (e.g., instances of connections, such as junctions, synapses, etc.)may be generated. The instantiation of connections may depend onposition tags and/or query results. The new data may be stored in thedatabase.

Tag implementation configuration may enable partitioning of the networksoftware application into two parts, which may include a data generationblock, a data storage block, and/or other parts. The data generationblock (e.g., implemented in c++) may be configured to generate databased on its own ‘intelligence’ and/or by interacting with the database(e.g., MySQL). In some implementations, the data generator functionalitymay be embedded within the database server. The data generator may beimplemented using server side procedures activated by triggers. Suchtriggers may include insert and connect call/trigger procedures storedon the database server.

In some implementations, instantiating an END synapse/junction mayrequire information such as one or more of class and ID of pre-synapticunit1; class and id of post-synaptic unit2; spatial position ofpre/unit1 and post/unit2, and spatial projection of pre/unit1.out andpost/unit2.in.; and/or other information.

A synapse/junction instance may be generated. In some implementations,additional external parameters may be used for instantiation of an ENDsynapse/junction. Examples of the external parameters may includesynaptic weights, synaptic delays, and/or other external parameters. Theuse and functionality of synaptic weights and/or delays in node-to-nodeconnections is described in further detail in a co-owned U.S. patentapplication Ser. No. 13/152,105, filed on Jun. 2, 2011, entitled“APPARATUS AND METHODS FOR TEMPORALLY PROXIMATE OBJECT RECOGNITION”and/or in a co-owned U.S. patent application Ser. No. 13/152,084, filedon Jun. 2, 2011, entitled “APPARATUS AND METHODS FOR PULSE-CODEINVARIANT OBJECT RECOGNITION”, each of the foregoing incorporated hereinby reference in its entirety.

To connect different network objects that use different spatialcoordinates, the coordinate system used for each network object may bepublished. That is, the coordinate system configuration may be availableto individual nodes within a certain scope.

Within the HLND framework, connections between network objects may beestablished in one or more ways. In some implementations, the connectionmay be established based on an overlap between the axon terminaldistribution and the synaptic bouton distribution. In someimplementations, the overall connection map may be obtained using ajoint probability distribution function (PDF) of the axon terminaldistribution and the synaptic bouton distributions. The joint PDF may beused to establish the required connections (synapses). In someimplementations, the HLND framework may be configured to distributeindividual ones of the potential connection points. Connection pointsmay be subject to one or more particular condition, such as, spatialcoordinate and/or other conditions. The HLND connection algorithm may beconfigured to select all (or a subset) of these connection points. TheHLND connection algorithm may be configured to instantiate thecorresponding connections. In some implementations, the HLND may beconfigured to generate arbitrary user-defined connection sets. Accordingto some implementations, the HLND may be configured to generateall-to-all connections.

SQL-Like Format

In some implementations, the HLND may implemented entirely using SQL.According to some implementations, such SQL implementation may beeffected using MySQL database and stored functions/procedures. The HLNDstatements may be constructed according to the English language grammar.

Tag Examples

As described above, individual network elements defined within the HLND,regardless whether it is a node, unit, synapse, or junction, or just anempty placeholder, may comprise a tag. This property of the HLND networkdescription may allow tagged elements to, inter alia, be addressed andmanipulated as a group. In some implementations, spatial coordinates maybe implemented using tags in the form (x,y,z).

By way of non-limiting example, a network unit may comprise one or moretags including the unit ID numerical identifier, ‘QIF’, ‘soma’,‘pyramidal’, ‘layer2/3’, ‘V1’, spatial coordinates tag (0.1,0.3,0.5),and/or other tags. A synapse may have tags such as, UD, pre_neuron,post_neuron, and/or other tags, denoting a pre-synaptic and apost-synaptic node IDs, respectively, ‘apical’, ‘exc’, ‘glu’, and aspatial coordinates tag (0.1,0.3,0.4).

Tagged Operator and Tag Filters

In some implementations, storing tags in a database may allow fastaccess to groups of elements. Individual database query statements thatoperate on tags may act as a tag filter (or a search statement) thatselects particular elements (that match the query terms) from thedatabase. For example, specifying <‘V1’> in the query, may result in aselection of individual elements that comprise ‘V1’ in any of its tags,e.g., the entire V1 subset. Specifying (<‘V1’ AND ‘pyramidal’ AND NOT‘layer2/3’) may result in individual pyramidal cells in V1 that are notlocated in the network layers 2 and 3.

In some implementations, an output of a tag query may be assigned itsown tag as follows:

Listing. 3 <tag filter> TAGGED new_tag

Some implementations may allow addressing of the elements that satisfythe tag filter, without copying and pasting the filter statement.

Example 3

-   The following statement:    -   exc OR inh TAGGED all        may add tag ‘all’ to all ‘exc’ and ‘inh’ neurons for easy        reference.

Example 4

-   The following statement:    -   (exc AND id<400) OR (inh AND id<100) TAGGED first_half        may cuts the network in half by assigning an extra tag to the        first half of elements.        OF Operator and Subsets

In some implementations, the expression

Listing. 4 n OF <tag filter>may return a list of n random elements satisfying the tag filtercondition. If the tag filter returns fewer than n elements, then someelements may be duplicated, so that the total number of elementsreturned by the expression of Listing 5 is equal to n. The OF operatormay not assign new tags. The OF operator may select a subset ofelements. To assign tags to the elements of the subset, the TAGGEDoperator may be used. The expression

Listing 5. 100 OF cones TAGGED S_conesmay select 100 elements from the node population of cones and may tagindividual selected elements as the S_cones. Similarly, the expressions

Listing 6. 300 OF (cones AND NOT S_cones) TAGGED M_cones cones AND NOTM_cones AND NOT S_cones TAGGED L_conesmay select 300 elements from the node population of cones (that are notin the subset S_cones), may tag individual selected elements as theM_cones, may select individual remaining elements from the nodepopulation of cones (that are not in either subset of S_cones orM_cones), and may tag each selected element as the L_cones.

Example 6

According to some implementations, a network comprising 800 excitory(exc) and 200 inhibitory (inh) neurons may be split into two equalsub-networks, subnetwork1 and subnetwork2, each comprising 400 exc and100 inh randomly selected neurons, as follows:

Listing 7. 400 OF exc TAGGED subnetwork1 100 OF inh TAGGED subnetwork1400 OF (exc AND NOT subnetwork1) TAGGED subnetwork2 100 OF (inh AND NOTsubnetwork1) TAGGED subnetwork2

Contrast the implementation of Listing 7 with the following statements:

Listing 8. 500 OF (exc OR inh) TAGGED subnetwork1 500 OF (exc OR inh)TAGGED subnetwork2.The statements of Listing 8 do not guarantee that the each ofsubnetwork1 and subnetwork2 comprises exactly 400 excitatory and 100inhibitory neurons.PUT Operator and Instantiation of Units

This PUT operator may be used to instantiate and tag network units asfollows:

Listing 9. PUT n OF unit_classThe PUT operator may be an instruction to create n instances of the‘unit_class’ type and tag them as (id, unit_class). Additional tags maybe subsequently added to these units by using TAGGED operator. The PUToperator may make calls to the respective constructor function of theunit_class to instantiate individual units. In the Listing 10, the OFkeyword may be used to cause generation of n copies of the unit_class bycalling the unit_class constructor n times.

Example 8

-   In accordance with some implementations, the statement

Listing 10. PUT 800 OF excmay create 800 units of exc class, with individual instances beingtagged as (id, exc).

-   The following statement

Listing 11. PUT 800 OF exc TAGGED exc_neuronscreates 800 units of exc class, with individual instances being taggedas (i) (id, exc); and/or (ii) additional tag exc_neurons, so thatindividual instances comprise two tags.

In some implementations, the PUT operator may be used to create unitinstances by using a filter parameter as follows:

Listing 12. PUT <tag filter> OF unit_classThe instruction of Listing 12 may be configured to create the number ofinstances of the unit_class corresponding to the number of elements thatare selected by the <tag filter> query field. Individual instantiatedunits in Listing 12 may be tagged with a tag corresponding to therespective element in the unit list that is selected by the query. Whenthe construction function, unit_class, is called, it may have access totags of the element that it instantiates (e.g., to the IDs, coordinates,etc.) so that constructor has sufficient information for unitconstruction.

Example 9

-   The statement

Listing 13. 800 ON circle(1) TAGGED my_points // see below thedefinition of ON PUT my_points OF excmay be configured to instantiate and distribute 800 units of exc classon a unit circle. The same outcome may be achieved by using thecomposite statement PUT 800 OF exc ON circle(1).CONNECT Operator and Connecting Units

The connect operator may be used to a synapse connection as:

Listing 14. CONNECT pre_tag TO post_tag WITH synapse_classwhere the parameter synapse_class denotes the class definition for thesynaptic connection and pre_tag, and the post tag denotes the filtermask specifying pre-synaptic units and post-synaptic units connected bythe synapses. In some implementations, multiple pre-synaptic and/orpost-synaptic units may be selected by the filter mask thereby resultingin the generation of multiple synaptic connections between the unitsthat satisfy the filter mask.

In some implementations, the synapse_class may be replaced by thejunction_class in the CONNECT statement so that the synaptic junctionsmay be generated. The construction function synapse_class may haveaccess to individual tags of pre-synaptic and/or post-synaptic elements.The construction function synapse_class may determine the delay and/orother relevant parameters needed.

Example 9

Some implementations may provide for the statements:

Listing 15. CONNECT N OF pre_tag TO post_tag WITH synapse_class CONNECTpre_tag TO N OF post_tag WITH synapse_classThe first statement in Listing 15 may be configured to generate aconnection matrix so that individual post_tag units are connected to Npre-synaptic units. The second statement in Listing 15 may be configuredto generate N outgoing synapses from individual pre_tag units to npost_tag units.

The statements in Listing 15 may use randomly selected subsets. This maybe implemented by a random walk through the list of all pre_tagged unitsand random selection of elements of the subset.

Example 10

Some implementations may provide for the statements:

Listing 16. n OF (CONNECT pre_tag TO post_tag WITH synapse_class)CONNECT pre_tag TO NEAREST post_tag WITH synapse_class CONNECT NEARESTpre_tag TO post_tag WITH synapse_class

The first statement in Listing 16 may be configured to instantiatesynapses comprising a random subset of the fully-connected matrix ofpre-to-post synapses. The term fully-connected matrix may be used todescribe a network configuration where all of the pre-synaptic units areconnected to all of the pus-synaptic units. Unlike the examples shown inListing 15, the first statement in the Listing 16 does not guaranteethat all pre-synaptic units or all post-synaptic units comprise the samenumber of synapses.

The second and/or the third statement in Listing 16 may be configured togenerate synaptic connections that are based on the coordinates ofpre-synaptic units and the post-synaptic. The second statement maycomprise a loop configured to connect each pre-synaptic unit to thenearest post-synaptic unit that satisfies the tag mask. The thirdstatement may loop through each post_tag and finds the nearest pre_tag.

In some implementations, the parameter NEAREST 1 OF may be used in placeof the parameter NEAREST in the statements of Listing 16.

In some implementations, individual pre-synaptic units may be connectedto n nearest post-synaptic units using the statement:

Listing 17. CONNECT exc TO NEAREST n OF exc WITH gluwhich may create ‘glu’-type synapses from individual excitatory neurons(i.e., a unit tagged ‘exc’) to n nearest excitatory neurons, includingitself (i.e., resulting in one auto-synapse).Generalized OF Operator

In some implementations, a generalized form of the OF selection operatormay be configured as:

Listing 18. <tag filter 1> OF <tag filter 2>The generalized OF operator may perform one or more of (i) creating alist_1 of all n elements that satisfy the tag filter 1 condition; (ii)creating a list_2 of all m elements that satisfy the tag filter 2condition; (iii) creating a list_3 by selecting (at random) a subset ofn elements from the list_2, if n>m then n-m elements are repeated in thelist_3; (iv) returning a merged list where each element from the list_1has an additional tag from the matching element in the list_3; and/orother actions. If the list_1 and the list_3 both comprise coordinatetags, then individual elements in the merged list may comprise acoordinate tag that is the sum of the coordinates of the correspondingelements, so that a single coordinate (x,y,z) tag per element may bemaintained.

Example 11

In some implementations, a set of units may be tagged with ‘cones’. Aset of random coordinates may be tagged with ‘retina’. Random retinalcoordinates may be assigned to the cones using the expression:

-   -   cones OF retina        When the number of cones is greater than the number of        coordinates, then multiple cones may be assigned the same        coordinates.        ON Operator and Assignment of Coordinates

The ON operator may be used to return a sample of n points from aprobability density function defined by the parameter pdf as:

-   -   n ON pdf,

Example 12

-   Some implementations may provide for the statements:

Listing 19. 1000 ON segment(0,1) TAGGED rnd 1000 ON circle(1) TAGGEDconesThe first statement in Listing 19 may be configured to generate a listof elements tagged ‘rnd’. The elements tagged ‘rnd’ may be distributeduniformly within the space segment defined by the coordinate x=[0 1].That is, elements may have tags (rnd, x) with the x-values uniformlydistributed in the range [0 1]. The second statement in Listing 19 maybe configured to generate a list of 1000 elements tagged as ‘cones’ anddistributed uniformly on a unit circle.

In some implementations, the ON operator may use individual pointsreturned by the function F:

-   -   ALL ON F        PER Operator and Combination of Tags

The operator PER may be used to iterate through a list of tagsspecified, e.g., by a tag filter. For individual elements of the list,the operator PER may call the statement, passing to it all the tags ofthe list element. The format of the PER operator may be:

-   -   statement PER tag_list

The PER operator may return a table comprising data describing generatednetwork elements. In some implementations, the PER operator may be usedto create multiple synapses per neuron. In some implementations, the PERoperator may be used to create multiple neurons per location. In someimplementations, the PER operator may be used to create multiplecortical columns per cortical surface.

Example 13

Some implementations provide for the statements:

Listing 20. 1000 ON segment PER neuron 1000 OF locations PER neuronSPNET

In some implementations, which may be applicable to SPNET, the unitclasses, exc and inh and the synaptic classes, glu and gaba may bedefined within the SPNET definitions.

Listing 21. PUT 800 OF exc PUT 200 OF inh CONNECT exc TO 100 OF exc ORinh WITH glu CONNECT inh TO 100 OF exc WITH gaba

The first line of Listing 21 may be configured to generate 800 units ofexc type. The second line of Listing 21 may be configured to generate200 units of inh type. The third line of Listing 21 may be configured toconnect individual units with tag ‘exc’ (note that the class type isautomatically can be used as a tag) to 100 randomly selected units withtags ‘exc’ or ‘inh’ with connection type glu. The fourth line of listing21 may be configured to connect individual units with tag ‘inh’ to 100randomly selected units with tag ‘exc’ with connection type gaba.

Retinal Pixel to Cone Mapping

In some implementations, the HLND description may be used to describeretinal pixel-to-cone mapping. Generally speaking, the cone cells, orcones, may be or refer to photoreceptor cells in the retina of the eyethat are responsible for color vision. Cone cells may be densely packedin the fovea, but gradually become sparser towards the periphery of theretina. Below several examples are provided that describe variousaspects of the retinal mapping.

Listing 22. // create a 100×100 square grid of pixel coordinatessquare_grid(100,100) TAGGED pixels //create pixel_units at each pixelcoordinate PUT pixels OF pixel_unit // create hexagonal grid of conecoordinates hexagonal_grid(100,100) TAGGED cones // mark a random subsetof 10% of them as S cones // and create appropriate S_unitsSIZE(cones)*0.1 OF cones TAGGED S_cones PUT S_cones OF S_unit // mark arandom subset of 30% of the remainder cones as M cones SIZE(cones)*0.3OF cones AND NOT S_cones TAGGED M_cones PUT M_cones OF M_unit // theremainder of cones are L cones cones AND NOT S_cones AND NOT M_conesTAGGED L_cones PUT L_cones OF L_unit // connect each pixel to onenearest S cone by a junction CONNECT pixels TO NEAREST S_cones WITHp2S_junction // the same to M cones CONNECT pixels TO NEAREST M_conesWITH p2M_junction // the same to L cones CONNECT pixels to NEARESTL_cones WITH p2L_junctionDirected Graph

In some implementations, assigning of the tag network subsets may beconfigured to enable representation of the network as a directed graph.A directed graph or digraph may comprise a pair G=(V,A) of elements,where a set V, whose elements may be called vertices or nodes, and a setA of ordered pairs of vertices, called arcs, directed edges, or arrows.In HLND, the term node may be used for vertices, and connections are theedges.

HLND and GUI

In some implementations, the HLND may comprise a Graphical UserInterface (GUI). The GUI may be configured to translate user actions(e.g., commands, selections, etc.) into HLND statements usingappropriate syntax. The GUI may be configured to update the GUI todisplay changes of the network in response to the HLND statements. TheGUI may provide a one-to-one mapping between the user actions in the GUIand the HLND statements. Such functionality may enable users to design anetwork in a visual manner by, inter alia, displaying the HLNDstatements created in response to user actions. The GUI may reflect theHLND statements entered, for example, using a text editor module of theGUI, into graphical representation of the network.

This “one-to-one mapping” may allow the same or similar information tobe unambiguously represented in multiple formats (e.g., the GUI and theHLND statement), as different formats are consistently updated toreflect changes in the network design. This development approach may bereferred to as “round-trip engineering”.

User Actions in HLND

In some implementations, the GUI may support user actions includingcreating nodes, selecting one or more subsets of the network, connectingnodes, tagging nodes within a selected subset, and/or other useractions. In some implementations, the GUI may support defining ofnetwork objects. Some exemplary user actions are described in detailbelow.

Create Nodes

Referring now to FIG. 12, node creation using GUI is illustrated, inaccordance with one or more implementations. In some implementations,creating nodes of the neural network may require information including atype of node to be instantiated and/or generated, a number of nodes tobe created, and/or other information. In some implementations, users mayprovide information including list TAGs to be assigned to the cratednodes, additional parameters for instantiating and/or initializingnodes, and/or other information. Additional parameters for instantiatingand/or initializing nodes may depend on a specific networkimplementation, such as, for example, instructions on how to lay out thenodes to be instantiated, that is, how to assign numerical spatial tags.

The above GUI Node Creation functionality may be supported by one ormore appropriate instructions of the HLND kernel that implement nodegeneration. See, for example, Listing 10, supra, for more detail. When auser enters the HLND node generation instruction (statement), the GUImay generate a graphical representation (e.g., a unique symbol,pictogram, and/or an icon) in the graphical editor corresponding to therespective node (or nodes). Entry of HLND statements by users may beeffected by various means, including but not limited to, text entry,speech, soft keys (icons), and/or other means configured for HLNDstatement entry.

A user may employ the GUI 1200 of FIG. 12 to perform node creation.According to some implementations, a user may select a node type fromavailable list of node types (the list 1210 in FIG. 12); drag and drop(as illustrated via the arrow 1204 in FIG. 12) the selected node type(e.g., the type 1216 in FIG. 12) into an editor panel 1202, where thenode is represented with a unique node symbol 1208; provide additionalparameters (if needed) in via a supplementary entry means (such as, forexample, a pop-up menu 1220 in FIG. 12) that is associated with thespecific node type 1216; and/or perform other actions to create a node.

The search box 1242 may allow the user to filter the list of displayednode types 1212, 1214, and 1216 using one or more keywords. This mayfacilitate node type selection where there are large number of nodetypes available. The pop-up menu 1220 may enable the user to graphicallyspecify a number of nodes 1226, parameters for the node instantiation1224, layout process 1230, additional tags 1232, and/or otherinformation associated with node creation.

The GUI may allow the user to switch back and forth between the texteditor (HLND statement 1240) and the GUI node creation. By way ofnon-limiting example, selecting different parameters/option for thelayout of the nodes in the GUI may update the HLND statement. Changingthe additional TAGs assigned to the nodes created in the HLND statementmay update this information in the GUI.

The GUI interface shown in FIG. 12 is not intended to be limiting asother implementations are contemplated and within the scope of thedisclosure. For example, in some implementations, the GUI may includepull down lists, radio buttons, and/or other elements.

Select Network Subset

Referring now to FIGS. 13, 13A, and 13B, different exemplaryimplementations of node subset selection are shown and described indetail. The GUI 1300 of FIG. 13 may comprise a network layout panel1302, two or more selection description panels 1304 and 1306, and/orother components. The network, shown in the panel 1302, may comprise aplurality of nodes with different tags 1305, 1308, and 1310, depicted as‘□’, ‘Δ’, ‘◯’, respectively. The selection description panels 1304 and1306 may comprise a Boolean portion of HLND statements, corresponding torespective subset.

In some implementations, Select Network Subset user action maycorrespond to selecting members of the network using the GUI editor(e.g., the GUI of FIG. 13). A user may select a subset of the network,for example, by using a mouse (or other pointing device, such astrackball, fingers on a touch-pad device like and iPad from Apple, lightpen, and/or other technology). The subset selection using GUI may beachieved via a selective clicking/tapping graphical symbolscorresponding to the desired members of the network, click and drag toselect an area of the network, a combination thereof, and/or otheractions to select a subset. The subset selection action of the GUI maybe supported by the respective instructions of the HLND kernel thatimplement subset selection. See, for example, listings 6-7, supra, formore detail.

As shown in FIG. 13, the node subset 1312 may comprise nodes comprisingthe tag 1305, while the subset 1314 may comprise nodes having both tags1308 and 1310. Once the subsets 1312 and 1314 are selected, the Booleanexpressions in the panels 1304 and 1306 may be updated accordingly.

In some implementations, the network shown in the GUI 1320 of FIG. 13comprises two subsets including the subset 1322 comprising tags 1305 and1308, the subset 1314 comprising the tag 1310, and/or other subsets. Theselection description panel 1324 may be updated to reflect the Booleanexpression corresponding to the tag content of the subset 1322. In someimplementations, additional subsets may be generated by forming a subset1326 comprising an intersect between the subsets 1322 and 1314, asindicated by the Boolean expression 1328.

By way of non-limiting example, responsive to a user entering theBoolean expression for the subset selection statement, e.g., theexpression 1306 in FIG. 13), the GUI may display (for example, using ashaded rectangle in FIG. 13) the corresponding selected members of thesubset in the graphical editor. In some implementations, the GUI maygenerate a graphical representation in the graphical editorcorresponding to the subset (or subsets) selection, as illustrated withrespect to FIGS. 13A-13B below. Examples of graphical representationsmay include one or more of a unique symbol, pictogram, an icon, and/orother graphical representations. In some implementations, a graphicalrepresentation may include a change in a graphical attribute including achange in one or more of color, shading pattern, and/or other graphicalattribute.

FIG. 13A illustrates an exemplary implementation of node subsetselection, which may be applicable to network subsets comprising largenumbers of nodes where depiction of individual nodes is not alwayspractical. The network shown in the GUI implementation 1330 of FIG. 13Amay comprise two subsets 1332 and 1334, which may be depicted usingdifferently shaded rectangles. The network shown in the GUIimplementation 1340 of FIG. 13A may comprise subsets, which may bedepicted by shapes with different fill patterns (see, e.g., rectangles1342 and 1344 in FIG. 13A). The subset 1346 in FIG. 13A may be selectedas 1342∩1344.

In some implementations, the GUI user actions may be represented usingunique symbols 1362 and 1368, as illustrated in network 1360 shown inFIG. 13B. The unique symbols 1362 and 1368 may represent subsets 1304and 1306, respectively, and may be subset specific. By way ofnon-limiting example, the color and/or other identifying quality of theunique symbol may be configured in accordance with the tags that areused to identify the subset. The shape and position of the symbol in thegraphical editor panel 1302 may be configured according with the spatialtags of the members of the subset. This may be illustrated by symbols1362, 1368, 1372, 1376, 1378, 1382, and 1388, which may depict subsetsof the network illustrated GUI implementations 1360, 1370, and 1380. Insome implementations, the symbol may be configured based on the elementtype. In some implementations, the symbol choice may depend on whetherthe subset comprises (i) only nodes, (ii) only connections; or (iii)nodes and connections.

In some implementations, the same network configuration (e.g., thesubset 1312 of FIG. 13) may be represented within the GUI graphicalpanel (e.g., the panel 1302 of FIG. 13) using different symbols/icons.In some implementations, which may correspond to a low-level detailnetwork view (corresponding to, for example, no or low zoom), the subsetmay be represented using a symbol (e.g., the symbol 1362 of FIG. 13B)without showing individual elements of the subset.

In some implementations, which may correspond, for example, to a networkwith limited processing capability or a network configured for batchupdates, the subset may be represented using a symbol without showingindividual elements of the subset.

In some implementations, which may be associated with a high-leveldetail network view (corresponding to, for example, to high zoom level,and/or when processing resources are available to process informationrelated to individual element of the subset), the subset may berepresented using graphical depiction providing further detail of thesubset (e.g., the representation 1312 of FIG. 13 that illustratesindividual subset elements at their appropriate locations.

In some implementations, the HLND framework selection operation may beperformed to assign additional tags to the selected members, to use theselected members (nodes in this case) in a connection statement, and/orto perform other actions.

The GUI may allow the user to switch back and forth between the texteditor (HLND statement) and the GUI subset selection. By way ofnon-limiting example, selecting different node members using the GUI maycause updates of respective the HLND statements. Changing the selectionin the text editor may update the selection in the graphical editor. Insome implementations, updating the selection may include highlightingand/or otherwise visually emphasizing the selected members.

Connect Nodes

The node connection user action may correspond to creating connectionsbetween network nodes. According to some implementations, in creatinginter-node connections, the HLND kernel may require one or more of afirst subset selection (e.g., a subset of nodes the connections willoriginate from), a second subset selection (e.g., a subset of nodes theconnections will terminate at), a connection type used to connect thefirst subset to the second subset, and/or other information associatedwith connecting nodes.

In some implementations, one or more additional parameters may beprovided to the HLND kernel including one or more of parameters forsetting the connectivity mapping (e.g., an all-to-all, one-to-one,one-to-nearest, and/or other user defined connection mappings),parameters for instantiating and/or initializing connection instances(e.g., initializing the synaptic weights), list TAGs to be assigned tothe created connection instances, and/or other parameters.

The HLND kernel may implement instructions configured to connect nodesin the network. (See, for example, listings 16-17, supra, for moredetail). By way of non-limiting example, when a user enters theconnection instruction, the GUI may create a corresponding graphicalrepresentation (e.g., draws a link/arrow from source to destinationselection of nodes) in the graphical editor to illustrate theconnections.

In accordance with one or more implementations, the user may use the GUIto select a source subset selection of network members, select adestination selection of network members, drag and drop the sourceselection onto the destination selection to connect first selection tothe second selection, and/or perform other actions. The GUI may generatelink/arrow elements in the graphical view representing the respectiveconnections between the source and the target members.

In some implementations, a pop-up menu associated with the connectionelements (link/arrow) may allow users to select a connection type fromthe available list of connection types. In some implementations, thepop-up menu may allow users to provide additional parameters forinstantiating and/or initializing connection instances. In someimplementations, the pop-up menu may allow users to set parameters forsetting the connectivity mapping.

The GUI may allow the user to switch back and forth between the texteditor (HLND statement) and the GUI connection creation. By way ofnon-limiting example, selecting different node members using the GUI maycause updates of the HLND statement that are associated with the nodedescription. Changing the selection in the text editor may update theselection (e.g., highlights the selected members) in the graphicaleditor.

FIG. 14 illustrates an exemplary implementation of using GUI forconnecting two node collections. The GUI 1400 may comprise the networknode view panel 1402, one or more node collection selection fields 1406and 1404, HLND statement fields 1442 and 1444, and/or other components.When node collections 1412 and 1414 are selected using, for example, amouse click-and-drag actions in the HLND GUI field 1402, the nodecollection selection fields 1404 and 1406 may be updated to reflectselected collections. As illustrated in FIG. 14, the collection 1412 maycomprise the nodes with the tag 1405 (depicted as ‘squares’). Thecollection 1414 may comprise the nodes with the tag 1408 (depicted as‘triangles’) and with the tag 1410 (depicted as ‘circles’). The nodecollection selection fields 1404 and 1406 may be updated to displayselected node with the tags corresponding to the collections 1412 and1414, respectively.

According to some implementations, additional tags may be assigned tothe collections 1412 and 1414 by invoking, for example, right click on aselection to assign new tags for the selected members of the network.The HDLN statement for the tagging may be automatically generated.

A supplementary graphical data entry means (e.g., the pop-up menu 1430in FIG. 14) may be invoked in some implementations using, e.g., a rightclick, as illustrated by the line 1418. The menu 1430 may be used to,inter alia, assign additional tags 1432 or new tags 1434 to the selectednodes.

In some implementations, by using drag-and-drop action, using a mouseand/or a finger on a touch pad device, illustrated by the arrow 1416 inFIG. 14 in the GUI field 1402, the first selection may be ‘dropped’ ontothe second selection, which may instruct the HLND engine to createconnections between the nodes 1405 of the collection 1412 and the nodes1408 and 1410 of the collection 1414.

Additional supplementary graphical data entry means (e.g., the pop-upmenu 1420 in FIG. 14) may be used to, inter alia, specify parameters forthe connections. Specifying parameters for the connections may includeone or more of setting connection type 1422, initializing parameters1424 for the connection type, specifying connectivity pattern 1426,assigning tags to the connections 1428, and/or other actions.

For one or more user actions performed with the node selections 1412 and1414 using the GUI 1400, corresponding HLND statements may beautomatically generated and displayed in the statement fields 1442 and1444, respectively.

The GUI interfaces shown in FIGS. 12-14 are not intended to be limitingas other implementations are contemplated with within the scope of thedisclosure. For example, some implementations may include pull downlists, radio buttons, and/or other components.

Relation of the HLND to the END Format

The HLND format may be designed to be compatible and/or to be used inconjunction with the Elementary Network Description (END) format, whichis described in the U.S. patent application Ser. No. 13/239,123 entitled“ELEMENTARY NETWORK DESCRIPTION FOR NEUROMORPHIC SYSTEMS” filed on Sep.21, 2011, incorporated supra. In some implementations, instances of ENDunits may be generated based on a HLND description (e.g., ENDimplementations of model neurons). Instances of END junctions and/or ENDsynapses may directionally connect the units. The HLND may define theanatomy, while the neural and synaptic dynamics may be defined in theapplied END classes. HLND may hide complexity and/or low-leveldifficulties of END, and may make network design a simple process.

The generated END instances may be used to generate a neural networkengine that implements and/or runs the specified model. That is, ENDinstances may be used to generate an engine that implements the networkdefined by the HLND description and/or the applied END classes. Theengine may be executed on arbitrary hardware platform from PC, FPGA, anyspecialized END compatible hardware, and/or other computer hardware.

FIG. 11 illustrates three basic structures of the END engine, which maybe implemented on a general purpose RISC/CISC central processing unit(CPU), graphics processing unit (GPU), in an integrated circuit (e.g.,an ASIC), and/or other processor. The structures of the END engine maycorrespond to the ‘unit’ 1101, the ‘doublet” 1111, and/or the ‘triplet’1121 in FIG. 11. The END engine may be configured to execute the unit,doublet, and triplet rules, and/or access to the memories of theseelements. The END formats may be treated as the hardware specificationlanguage that would configure a semiconductor circuit having such units,doublets, and triplets that executes a specified neuronal network.

In some implementations, individual basic structures (e.g., unit,doublet, and/or triplet) may be implemented as a single thread on amulti-thread processor. In some implementations, individual structuresmay be implemented as a super-unit, super-doublet, and/or super-triplet,which may comprise dedicated circuits configured to processing units,doublets, and/or triplets respectively using time multiplexing. Someimplementations may include three different circuits, one for each ofunits, doublets, and triplets.

In some implementations, unit 1101 may represent a neuron and/or a partof a neuron (e.g., a dendritic compartment). In another example, unit1101 may represent a population of neurons. The activity of the neuronmay represent a “mean-firing rate” activity of the population and/orsome other mean-field approximation of the activity of the population.Individual units may be associated with memory variables and an updaterule that describes what operations may be performed on its memory. Theoperations may be clock-based (i.e., executed every time step of thesimulation) or they may be event-based (i.e., executed when certainevents are triggered).

Depending on the values of the unit variables, the units may generateevents (e.g., pulses or spikes) that trigger synaptic events in otherunits via doublets. For example, a unit 1102 in FIG. 11 may influenceunit 1103 via the doublet 1111, which may represent a synapse frompre-synaptic neuron (pre-synaptic unit 1102) to post-synaptic neuron(post-synaptic unit 1103).

Individual units may have after-event update rules, which may betriggered after the event is triggered. These rules may be responsiblefor modification of unit variables that are due to the events, e.g., theafter-spike resetting of voltage variables.

Individual doublets may be associated with memory variables. Individualdoublets may access variables of the post-synaptic unit. Such access mayinclude read, write, and/or access mechanisms. Individual doublets maybe associated with a doublet event rule that makes a change to thedoublet memory to implement synaptic plasticity. Individual doublets maybe associated with a doublet event rule that makes a change to thepost-synaptic unit memory to implement delivery of pulses. A doubletevent rule may encompass some or all the synaptic rules described in theEND formats above.

Because multiple doublets (e.g., 1116-1118 in FIG. 11) may connectcorresponding multiple pre-synaptic units 1106-1108 to a singlepost-synaptic unit 1109, the doublets may modify the post-synaptic unitmemory in parallel and/or in arbitrary order. The result may beorder-independent. This may be achieved when the operation on thepost-synaptic unit memory is atomic addition (as in GPUs), atomicmultiplication (which is equivalent to addition via logarithmictransformation), and/or resetting to a value (with all the doubletstrying to reset to the same value). The post-synaptic unit variable thatis being modified by the doublet event rule may not be used in the rule.Otherwise, the result may depend on the order of execution of doubletevent rules.

Referring now to FIG. 15, an exemplary implementation of a neuralnetwork definition system comprising HLND kernel and END description isshown and described in detail. In FIG. 15, the circles 1504, 1502, and1506 may represent different higher level network description methods orformats. The circle 1510 may represent the END description of thenetwork. Arrows from 1504, 1502, and 1506 may denote the process ofconversion to the END description. For example, a software thatprocesses the HLND description of a network (e.g., the HLND statements)may generate the END description of the same network. The rectangles1512, 1515, 1516, and 1518 in FIG. 15 may denote various hardwareplatform implementations of the network defined by the END description1510. The arrows between the circle 1510 and the rectangles 1512, 1515,1516, and 1518 may denote the engine generation process. By way ofnon-limiting example, the arrow between the END description 1510 and therectangle 1512 may represent the process of generating an executablethat implements the END network and is configured to runs on a CPU. TheHLND definition 1502 may be processed and converted into the ENDdescription 1510. The END description may be configured to be processed(e.g., by various software applications) to generate platform specificmachine executable instructions. The platform specific machineexecutable instructions may be configured to be executed on a variety ofhardware platforms, including but not limited to, elements generalpurpose processor 1512, graphics processing unit 1514, ASIC 1516, FPGA1518, and/or other hardware platforms.

Other network description formats may be used with the process 1500,such as, for example BRIAN 1504 and/or other neuromorphic networkdescription format 1506 (e.g., NEURON) configured to generate the ENDdescription of the network, as illustrated in FIG. 15.

Exemplary Implementation of Computerized HLND Apparatus

An exemplary implementation of a computerized network processingapparatus configured to utilize HLND framework in designing a neuralnetwork (e.g., the network 1500 of FIG. 15) is shown and described withrespect to FIG. 16. The computerized apparatus 1600 may comprise aprocessing block (e.g., a processor) 1602 coupled to a nonvolatilestorage device 1606, random access memory (RAM) 1608, user input/outputinterface 1610, and/or other components. The user input/output interfacemay include one or more of a keyboard/mouse, a graphical display, atouch-screen input-output device, and/or other components configured toreceive input from a user and/or output information to a user.

In some implementations, the computerized apparatus 1600 may be coupledto one or more external processing/storage devices via an I/O interface1620, such as a computer I/O bus (PCI-E), wired (e.g., Ethernet) orwireless (e.g., WiFi) network connection.

In some implementations, the input/output interface may comprise aspeech input device (e.g., a microphone) configured to receive voicecommands from a user. The input/output interface may comprise a speechrecognition module configured to receive and recognize voice commandsfrom the user. Various methods of speech recognition are consideredwithin the scope of the disclosure. Examples of speech recognition mayinclude one or more of linear predictive coding (LPC)-based spectralanalysis algorithm run on a processor, spectral analysis comprising MelFrequency Cepstral Coefficients (MFCC), cochlea modeling, and/or otherapproaches for speech recognition. Phoneme/word recognition may be basedon HMM (hidden Markov modeling), DTW (Dynamic Time Warping), NNs (NeuralNetworks), and/or other processes.

The END engine 1510 may be configured to convert an HLND description ofthe network into a machine executable format, which may be optimized forthe specific hardware or software implementation. The machine executableformat may comprise a plurality of machine executable instructions thatare executed by the processing block 1602.

It will be appreciated by those skilled in the arts that variousprocessing devices may be used with various implementations, includingbut not limited to, a single core/multicore CPU, DSP, FPGA, GPU, ASIC,combinations thereof, and/or other processors. Various user input/outputinterfaces may be applicable to various implementations including, forexample, an LCD/LED monitor, touch-screen input and display device,speech input device, stylus, light pen, trackball, and/or other userinterfaces.

Execution of GUI User Actions

In some implementations, the network design system, e.g., the system1600 of FIG. 16, may automatically convert the GUI actions into HLNDinstructions and/or into END statements. The HLND instruction may causeautomatic updates of the GUI representation and/or of the ENDdescription.

FIG. 17 illustrates one approach to performing seamless updates ofdifferent representations corresponding to the same network designelement. The network description 1702 (e.g., node, connection, subset,etc.) may contain information necessary to define the network. In someimplementations, the network description (1702) may comprise one or moreof node type, node type parameters, node layout parameters, tags, and/orother information. In some implementations, the network description(1702) may comprise one or more of connection type, connection typeparameters, connection pattern, tags, and/or other information: Variousother description types may exist (e.g., subsets), which may compriseappropriate information associated therewith.

As illustrated in FIG. 17, a single object (e.g., the object 1702) mayhave one or more representations related thereto, which may include GUIrepresentation 1712 (e.g., using the GUI editor 1302 of FIG. 13), theHLND representation 1714 (e.g., using the HLND statement described suprawith respect to FIG. 14), the END representation 1716 (see, e.g., FIG.15), and/or other representations (depicted by the rectangle 1718).Responsive to an object property (i.e., the object data element) beinggenerated and/or updated, the corresponding representations of theobject (e.g., the representations 1712, 1714, 1716, and 1718) may beupdated using bi-directional pathways 1720, 1722, 1724, and 1726,respectively.

In some implementations, responsive to a user GUI action modifying aselection, the corresponding HLND statement(s) (e.g., the HLNDrepresentation 1714 in FIG. 17) may be updated. In some implementations,the END instructions (e.g., the END representation 1722 in FIG. 17) maybe updated.

In some implementations, the END instructions may be executed by theapparatus thereby enabling a more detailed and accurate representationof the network.

In some implementations, nodes may be rendered within the GUI usingunique color symbol when the statement to create units is available inthe network description framework.

In some implementations, nodes may be rendered within the GUI at theircorrect location with unique (for the whole subset) symbols responsiveto the coordinates of the nodes being available—that is, when theconnections statement is at least partially processed.

In some implementations, connections between two subsets may be renderedby the GUI using, for example, a single line, when the connectinstruction is available in the network description framework.

In some implementations, connections between two subsets may show thedetailed connectivity structure once the pre-node and post-nodeinformation for the connection instances are available (i.e., have beenpreviously generated)—that is, once the connection statement is at leastpartially processed.

In some implementations, connections between two subsets may show thedetailed connectivity structure with unique properties (e.g., line widthrepresenting a connection) per connection responsive to the pre-nodeinformation, the post-node information, and/or the initial weight forthe connection instances being available—that is, once the connectionstatement is at least partially processed.

As is appreciated by those skilled in the arts, other representations(e.g., as depicted by the rectangle 1718 in FIG. 17) may exist and maybe compatible with various implementations, provided they conform to theupdate framework described herein.

In some implementations, data exchange between different representations(e.g., the representations 1712, 1714, 1716, and 1718 in FIG. 17) may beenabled via direct links denoted by arrows 1730, 1732, 1734, 1736, 1738,and 1739 in FIG. 17. For clarity, not all direct connections between therepresentations 1712, 1714, 1716, and 1718 are shown in FIG. 17.

It will be recognized that while certain aspects of the disclosure aredescribed in terms of a specific sequence of steps of a method, thesedescriptions are only illustrative of the broader methods of thedisclosure, and may be modified as required by the particularapplication. Certain steps may be rendered unnecessary or optional undercertain circumstances. Additionally, certain steps or functionality maybe added to the disclosed implementations, or the order of performanceof two or more steps permuted. All such variations are considered to beencompassed within the disclosure disclosed and claimed herein.

Although the disclosure has been described in detail for the purpose ofillustration based on what is currently considered to be the mostpractical and preferred implementations, it is to be understood thatsuch detail is solely for that purpose and that the disclosure is notlimited to the disclosed implementations, but, on the contrary, isintended to cover modifications and equivalent arrangements that arewithin the spirit and scope of the appended claims. For example, it isto be understood that the present disclosure contemplates that, to theextent possible, one or more features of any implementation can becombined with one or more features of any other implementation.

What is claimed:
 1. Computer readable apparatus comprising anon-transitory storage medium, said storage medium comprising at leastone computer program with a plurality of instructions, said at least oneprogram being capable of implementing round-trip engineering method in aneural network by at least: enabling receiving user inputs via agraphical user interface; based at least in part on first user inputindicative of a plurality of nodes within said network, automaticallygenerating first single network description instruction; based at leastin part on second user input indicative of a tag, automaticallygenerating second single network description instruction; and based atleast in part on third user input indicative of node connections,automatically generating third single network description instruction;and wherein: said first, second and third single instructions, whenexecuted, are configured to: (i) generate said plurality of nodes; (ii)select a portion of said plurality of nodes based on said tag, and (iii)generate a plurality of connections associated, at least in part, withsaid portion of said plurality of nodes, substantially without relyingon subsequent user input.
 2. The apparatus of claim 1, wherein saidplurality of connections is distributed within the network in accordancewith a probability distribution comprising a spatial parameter.
 3. Theapparatus of claim 1, wherein said method further comprises assigningone other tag to said plurality of connections, said one other tag beingdistinct from said tag.
 4. The apparatus of claim 1, wherein said eachsaid first, second and third single instruction is configured consistentwith English language structure and grammar.
 5. Computer readableapparatus comprising a non-transitory storage medium, said storagemedium comprising a plurality of computer instructions being capable ofimplementing round-trip engineering (RTE) method in a computerizedneural network, the method comprising: receiving first user inputindicative of generation of a plurality of nodes within said network;and based at least in part on said input, generating a single networkdescription instruction, which, when executed, causes generation of saidplurality of nodes within said network, substantially without furtheruser input.
 6. The apparatus of claim 5, wherein: the first user inputis having a graphical representation of said network associatedtherewith; and the method further comprises: receiving second user inputcomprising one other single network description instruction; and basedat least in part on said second input, automatically updating, inaccordance with said on other single instruction, said graphicalrepresentation of said network thereby effecting said RTE.
 7. Theapparatus of claim 5, wherein: the first user input is having agraphical representation of said network associated therewith; and themethod further comprises: receiving second user input comprisingmodifying said single network description instruction; and based atleast in part on said second input, automatically updating, inaccordance with said modified single network description, said graphicalrepresentation of said network thereby effecting said RTE.
 8. Theapparatus of claim 5, wherein said plurality nodes is characterized by aprobability distribution having a parameter associated therewith.
 9. Theapparatus of claim 8, wherein said parameter comprises spatial extentwithin said network, and generation of said plurality of nodes comprisesdistributing said plurality nodes within said network in accordance withsaid probability distribution with said spatial extent.
 10. Theapparatus of claim 5, wherein said single instruction compriseshigh-level network description (HLND) instruction comprising a keywordselected from the group consisting of CREATE, MAKE, and PUT.
 11. Acomputer readable apparatus comprising a non-transitory storage mediumadapted to store a computer program, said computer program which, whenexecuted, causes creation of a plurality of probabilistic connections ina neural network comprising a plurality of nodes by: based at least inpart on receiving user input via a graphical user interface,automatically generating a single network description instruction, saidsingle instructions configured to generate said plurality ofprobabilistic connections.
 12. The apparatus of claim 11, wherein eachof said plurality of probabilistic connections comprises one of (i)synapse; and (ii) junction.
 13. The apparatus of claim 11, wherein saidplurality of probabilistic connections comprises connections betweenfirst portion of nodes and second portion of nodes of said plurality ofnodes; and nodes of said first portion are being randomly selected fromsaid plurality of nodes.
 14. The apparatus of claim 13, wherein computerprogram is further configured to: based at least in part on said userinput, generate first representation of the network; and based at leastin part on said single network description instruction, generate secondrepresentation of the network; wherein, said first representationmatches said second representation thereby effecting round-tripengineering of said network.
 15. The apparatus of claim 14, wherein saidfirst representation comprises representation of said first portion andsaid second portion; and said second representation comprises, at leasta part of, said single network description instruction.
 16. Theapparatus of claim 13, wherein said randomly selected nodes arecharacterized by a probability distribution having a parameterassociated therewith.
 17. The apparatus of claim 16, wherein parametercomprises an alphanumeric tag associated with nodes of said network. 18.The apparatus of claim 16, wherein parameter comprises a spatialcoordinate associated with nodes of said network.
 19. The apparatus ofclaim 11, wherein said plurality of probabilistic connections comprisesconnections selected randomly from all possible/attainable connectionsbetween first portion of nodes and second portion of nodes of saidplurality of nodes.
 20. Computer readable apparatus comprising anon-transitory storage medium, said storage medium comprising aplurality of instructions, said plurality of instructions comprisinginstructions configured to implement round-trip engineering of a neuralnetwork, comprising a plurality of nodes, by: receive, via a graphicaluser interface, user input indicative of a portion of said plurality ofnodes; and based at least in part on said input, generate a singlenetwork description instruction configured to assign a tag to saidportion.
 21. The apparatus of claim 20, wherein assigning said tag tosaid portion comprises assigning said tag to each node of said portion.22. The apparatus of claim 20, wherein: said user input corresponds tofirst representation of the network; said single network descriptioninstruction corresponds to second representation of the network; andsaid first representation matches said second representation therebyeffecting said round-trip engineering.
 23. The apparatus of claim 20,wherein said tag comprises a sting identifier.
 24. The apparatus ofclaim 20, wherein said tag comprises an alphanumeric identifierconfigured to identify said portion.
 25. The apparatus of claim 24,wherein said alphanumeric identifier is adapted to indicate spatialcoordinate of nodes within said portion.
 26. The apparatus of claim 20,wherein said single instruction is configured consistent with Englishlanguage structure and grammar.
 27. The apparatus of claim 20, furthercomprising executing said single instruction by a computerized apparatusthereby assigning said tag to each node of said portion; wherein saidportion is characterized by a probability distribution, comprising aparameter indicative of a spatial coordinate; and nodes of said portionare being distributed within said network in accordance with saidprobability distribution.
 28. A system configured to implementround-trip engineering in a neural network, the system comprising: oneor more processors configured to execute one or more computer programmodules, wherein execution of individual ones of the one or morecomputer program modules causes the one or more processors to: receive afirst user input indicative of generation of a plurality of nodes withinthe neural network; and generate a single network descriptioninstruction based on the first user input, the single networkdescription instruction being executable to cause generation of theplurality of nodes within the neural network substantially withoutfurther user input.
 29. A system configured to implement round-tripengineering in a neural network, the system comprising: one or moreprocessors configured to execute one or more computer program modules,wherein execution of individual ones of the one or more computer programmodules causes the one or more processors to: receive, via a graphicaluser interface, user input indicative of a portion of the plurality ofnodes; and generate a single network description instruction based onthe user input, the single network description instruction beingconfigured to assign a tag to the portion.